Prior to joining Intercom as a Product Manager, I had never run a structured beta. When it came to finally running my first one, I was surprised to find very little information online that could help me.
I’ve run a lot of successful betas now but I learned my craft through tribal Intercom knowledge, built up by other Product Managers over the years.
The content I did find online was overly prescriptive and too focused on the specifics that weren’t going to be relevant to a wider audience with different products. To address that deficit, I’ve put together a guide on how we run betas here at Intercom.
Short on time? Here are the key takeaways:
Understand the problem that you’re trying to solve This will pinpoint why you’re running a beta and help to define what you want to learn as early as possible. This should be done before you even create a build plan.
Paint a picture of what success looks like Decide how you will measure it so you can assess whether or not you are solving the problem adequately.
Consider who should get access Start small with early adopters who already experience the problem you’re trying to solve. Broaden the audience over time to include users who haven’t asked for the feature.
Decide how you’re going to pitch Work on how you’ll pitch and describe the solution in a way that makes the most sense for your users.
Get feedback direct from your customers Jump on customer calls and send targeted messages based on usage of the beta to gather feedback. Create one source of truth that keeps this up to date and organized.
Evaluate the findings You should evaluate all of your learnings and feedback against the problem you set out to solve. Decide whether or not you are holding something of value back from non-beta customers or if it falls short of the bar you set out to exceed.
Decide when to ship Set a timeline to make a final call on whether and when you will ship the feature you’re testing – this is a crucial forcing function to avoid becoming paralyzed by the mountain of feedback.
Step 1: Why should you run a beta?
One of the biggest pitfalls Product Managers continuously make is planning a beta as the feature approaches customer readiness. That’s too late – you’re likely asking the question “Will we beta this?” because you feel you should be. Instead, asking yourself, “What do I want to learn as early as possible?” should be your starting point.
Start with why and everything else will fall into place.
For any project that contains uncertainty you should be considering whether you will release an early version to a group of users from the outset. At Intercom, we scope small. It can feel uncomfortable but there’s always a version that can be released early that you can begin to learn from.
The answer to “What do I want to learn as early as possible?” will inform how you roll out the final feature, not just your beta. Start with why and everything else will fall into place, like who will get access, when do they get access, and what they will get access to.
Not every feature you build will require a beta. We have shipped new functionality that didn’t allow for or justify the overhead of running a beta. An example of this was enabling customers to capture an email and replace the standard conversational reply on outbound messages.
Our new feature required considerable volume and we were working under tight timelines so we decided it would be better to release and learn rather than run a half-baked beta. As a form of QA, we worked alongside our Content team to dogfood the new feature on our blog and see how it performed in a short period to decide whether it was worthy of release or not.
Betas are not for finding bugs. Betas can be a great safeguard against bugs but it’s dangerous to release things to beta that you are not confident in your QA efforts on.
Step 2: What does a successful beta look like?
You need to define what success will look like for you by outlining what metrics are relevant and need to be measured. You should also set targets for these as a benchmark.
Success metrics are essential for comparing progress to.
Success metrics are essential to having something tangible in place to compare progress to, inform actions, provide a mechanism to fight for during the beta and post release, and prompt important questions.
When the new reports for our Inbox product were in beta, we gave users the ability to switch between the old-style reports that were available to all of our customers and the new reports that were only available to those in the beta.
We had adoption and engagement metrics in place with targets we wanted to hit. These metrics suggested a large cohort of customers switched directly back to the old reports on every login. To learn the real reason why this was, we decided to remove access from the old reports once we were confident that the new reports were doing what they should.
We learned two things very quickly from this:
The majority of customers switched back to the old reports out of habit
Two specific insights our old reports were enabling in a better way were being catered for poorly in our new design
We iterated on our new reports design and then were in a much better place to rollout and replace reports in a more informed and confident way.
Step 3: Who should you invite to your beta?
You need early adopters who are likely to get started with the new feature or product right away and be all right with something that could be a little rough around the edges. To find these people, we start with conversation records of customers who have requested the feature or experienced the problem we aim to solve.
Your solution should be the only incentive needed.
You should always start with a small rollout and then increase the audience based on adoption and feedback. Unless there is a risk or downside to the beta, such as breaking a primary workflow, you should opt in apps and notify them, rather than asking if they want to be included. Asking if they want to be included is just an additional step that creates a hurdle. Of course, this varies wildly as a rule of thumb depending on your business.
As your beta matures, add a random selection of other users who haven’t experienced the issue or requested the feature to ensure you aren’t only receiving feedback from a biased group.
Incentives are built in to many beta programs but if you feel like you need one to get usage, you should ask yourself two things: Do you know who needs this? And does anyone need this?
The solution should be good enough as an incentive in and of itself or else something is wrong. We scope our betas to include at least some customer value, otherwise where is the hook?
Step 4: How to communicate your beta to your customers?
When it comes to beta messaging at Intercom, we rely primarily on messages sent via our own Intercom messenger. We use tags to manage beta participants and send messages that are contextual to them being in the product. The other major benefit is that customers can start conversations, ask questions and provide feedback via this message.
Here’s what you should include in your message:1. Stress to the customer that it is a beta in order to set expectations.2. Describe the problem.3. Describe what the solution is and where they can find it. Be visual.4. State that you need feedback in order to improve the feature and are building the most impactful solution.
Step 5: How to get feedback on your beta
Specifying that you’re running a beta in order to capture feedback should be enough to kickstart organic feedback loops. People who offer feedback without being prompted are invaluable. You shouldn’t underestimate the importance of jumping on calls with these customers early on. You don’t need to conduct state of the art user research sessions, just ask for a few minutes of their time.
Be careful not to burn bridges with customers.
Before the call, you should review what they have and haven’t done then jot down the qualitative questions you care about them answering the most. Invite an engineer and designer involved in the work to listen in and make sure to capture every piece of feedback.
You can’t rely solely on proactive feedback. Usage metrics are crucial for feeding into whether the project is a success or not and will help inform the questions you ask. Sending a message requesting feedback to people who don’t use the feature is a waste. Only a small set of customers will use the feature and provide quality feedback so it’s important not to burn bridges. Targeting your messages is key to getting the most out of feedback opportunities and keeping on the good side of your beta users.
Step 6: Making sense of beta feedback
It’s crucial to capture every insight and information point during the beta process. Product Managers often fall into the trap of tracking some feedback on Post-its, lodging others in their brains’ feedback depository and spreading the rest between different scraps of paper and Google docs. You’ll hate yourself when it comes to crunch time and you need to make decisions.
One of the best ways to consistently track all of the qualitative feedback is to create a spreadsheet before ever adding a beta customer. Remain disciplined about adding EVERYTHING in here, even duplications of feedback or questions about the beta.
You can then codify your feedback to make it as useful as possible. Below is an example of how to codify beta feedback:
Column 1 – Original feedback or comment
Column 2 – Type (e.g. feature request, blocker, comprehension, usability, improvement)
Column 3 – Summary. This is my concise version of their feedback
Column 4 – Reason. The issue, cause or problem actually being expressed
Once you’ve gathered a significant amount of data, this method of storing feedback will help you easily communicate the patterns and findings to anyone who is interested.
When it comes to dealing with feedback, there are some core principles to keep in mind:
Always chase for the why
The majority of feature requests should be considered and acted on post release, unless there is a worrying trend and the problem you set out to face wasn’t actually the most important problem
Look out for patterns of the same question – either the UI or logic is confusing, or this will help inform what info product education docs should provide
Step 7: Is your feature ready to ship?
Your success metrics and qualitative feedback will ultimately dictate what and when you should release. However, setting a timeline for when you will make that all-important call is great as a forcing function for actions. This can vary depending on the complexity of what you are betaing and the certainty of the solution, so there’s no rule of thumb here.
Set yourself a timeline as a forcing function.
When we launched the new Respond Reports from beta, there was a mountain of feedback. In an ideal world, it would have been great to bundle these into what we launched. However, this would have delayed replacing an existing report with a better reporting feature for the 95% of customers not in the beta. So it made sense to release what we had and benchmark the other requests and feedback for iteration and improvement in the near future.
Before making your final decision, you should step back and ask yourself some questions:
Do you feel confident that you have solved the problem enough to warrant releasing your feature? Use measures of success and a general sense of qualitative feedback to guide you.
Is it replacing something that exists already? If it’s better than what is currently available to everyone, you should probably ship it as soon as possible.
Does your marketing team want to announce at a particular time or in a particular way?
As with most processes, how you run your betas will evolve with your business. At Intercom, we’re always learning and iterating with each additional feature.
Comments