What are you hoping to achieve when you message customers? Do you want them to simply open the email, and maybe click a link? Or do you actually want them to take some real action in your product?
Magoosh was one of our earliest and heaviest users of Intercom’s Message Goals feature while it was in beta.
Magoosh is a web and mobile application that helps students prepare for standardized tests. The application allows students to learn at their own pace, on their own time, and in an adaptive assessment environment.
What follows is a post by Kevin Rocci, on how Magoosh use Message Goals and A/B testing to successfully get more students to engage with their learning materials.
Let’s be frank – standardized testing is not fun. Our perception of preparing for one of these tests usually conjures images of thick expensive books, Red Bull fueled 8-hour study sessions, long classes with instructors of varying skill, or a painfully solo journey of rote memorizing.
Magoosh exists to change all that. We want to shatter those expectations by offering materials and products that are accessible, effective, and engaging. We believe that learning can be fun, even if it’s preparing for a standardized test.
The core value of meaningful data
We value meaningful data. One of our core values is Data > Intuition, so we knew that standard metrics — like email open rates and click through rates — weren’t enough to tell us if we were successful. Those numbers are easy to game, and they don’t achieve the real goal: to help students learn.
Since we have a healthy culture of experimenting and measuring success towards goals with metrics, we instantly gravitated to the simple and easy A/B testing and Goal-setting features in Intercom. They’ve become a regular part of any message we send to our students because it’s so easy to track metrics. It also ensures that any message we send will be meaningful and effective. We’ve gotten to the point where if we don’t A/B test or don’t have a goal set, we re-evaluate why we are even sending the message.
For example, we help high school students prepare for the Standardized Achievement Test (SAT), a common requirement on university applications. We noticed high schoolers tended to not be a strongly self-motivated bunch, so our goal for the quarter was to increase engagement by encouraging students to answer more online questions and watch more interactive lessons.
We created an Intercom message to send out to students who had been studying, answering questions, watching lessons, but stopped. After 5 days of inactivity, they would get a message that would encourage them to continue studying.
But how do you motivate a high schooler to study? What sort of message should you write? What should you talk about? Having the A/B Test and Goal features in Intercom made it easy to experiment with a message that really motivated students.
Messages that motivate
We started our A/B test with two messages: one that focused on getting back to studying for the SAT and one focused on getting to college. We had a hunch that students would be more motivated to study if we tied success on the SAT to college and life goals—not success on the SAT itself. So we crafted one message to link success on the SAT to larger life goals. In the message, we said:
The other message focused on studying and preparing in order to succeed on the SAT:
Our goal was for students to login and practice so we added a Goal for ‘web sessions’ to increase and set it live.
After letting the experiment run for a few weeks, we found that the message that tied success to future goals won. It was nice to see that one message had a better open rate than the other — 57% to 60% — but the real win was the seeing more students log in and study.
With Goals, it was easy to see that the message about life goals had 236 students logging in and studying versus 227 students for the other message. This doesn’t look like a significant difference—a difference of 9 between the two tests. How could this be a win?
We use isvalid to help. This tool allows us to quickly check the significance of our numbers that we get back from the goal feature in Intercom. We want to know if one message is directionally better than another. This allows us to turn off the test and move to the next one quickly. We found that we were at 95.8% significance with our message with the winner performing 11% better than the other. This was good enough for us to turn off the experiment and choose a winner.
Goals and A/B testing make it so easy for us to commit to our core value of Data > Intuition. Not only can we default to testing and experimenting to find what really works, but also Intercom makes the features very easy to use so new team members at Magoosh can jump in right away and start testing.
Comments