How to optimize your server with A/B Testing

Reading Time: 4 minutes

A/B Testing is one of the most powerful tools to increase revenue per user and user retention, which is why every Fortune 500 company uses it. With MCMetrics, you have access to the same technology in order to optimize your revenue per player and player retention in a matter of minutes.

How A/B Testing Works (and why do it?)

This is best explained with an example:

1. You figure out that your server's onboarding process (what the player sees when they join for the first time) is not optimized through MCMetrics Player Retention Analysis, Custom Queries, etc.)
2. You examine the current onboarding flow on your server. In this example, it's a chat message welcoming the player and telling them to run /island create to get started.
   1. This is your control variant (for later).
3. You come up with some better ideas for an onboarding process:
   1. An interactive tutorial walking the player through the best commands
   2. Automatically creating the island for the player and teleporting them there
4. Problem: between these 3 variants of onboarding processes (control, a, b), which one should you use? This is important, because if you don't implement the best variant, you will lose out on players (= revenue)
5. Solution: Test all 3 on real users with MCMetrics A/B Testing! Create an A/B test on MCMetrics that looks like this:

Click Create. MCMetrics will now handle the rest for you:

• The A/B test will be triggered for any player that joins for the first time on your main server
• This new player will have a 60% chance of seeing the control onboarding flow, 20% chance of the interactive tutorial, and 20% chance of automatically creating the island
  • The MCMetrics plugin will randomly assign one of these variants to the player when they join for the first time
• Then, it will execute the appropriate command:
  • If it's the control variant, do nothing (No action) because the player will already receive the welcome message automatically
  • If it's the interactive tutorial, the console command start-interactive-tutorial ${player} will start the tutorial for the player (${player} will be replaced by the player's username)
  • If it's the auto-create island variant, the plugin will make the player create their own island

You can now click "View A/B Test" on the MCMetrics dashboard at any time:

This will open up the results page, which might look like this:

On this page, MCMetrics will tell you how many players have seen the A/B test and how many it recommends:

The participant goal is calculated automatically. If it's not reached, there might not be enough data to be sure whether or not a variant is best

At the bottom of the page, MCMetrics will tell you exact metrics for each variant:

In this case, you can see that the interactive tutorial has the highest ARPU (Revenue per user) and retention by far.

Best Variant

(New in MCMetrics V2)

To save you time, MCMetrics will also tell you which variant is likely the best and recommend an action for you to take. In this case:

The interactive tutorial has a 92.5% chance of being the best and is the top result.

(Side Note: this 92.5% chance is calculated using a Bayesian simulation with 100,000 iterations. Read more here)

Since there is a clear winner variant and we have enough data, MCMetrics will give you a simple recommendation:

At this point, it makes sense to send 100% of players to the interactive tutorial since you have proven that it is the best tutorial. MCMetrics will even tell you how big of a difference this will make:

+324 retained players and +$15,750 in revenue in this example thanks to MCMetrics + A/B Testing 💪

Now that you know what an A/B test is, how to do one with MCMetrics, and why it's so powerful, create some on mcmetrics.net! Large companies often run hundreds or thousands of A/B tests at the same time. Think about it like this: Even if an A/B test only boosts your revenue by 1%, that could add up to hundreds or thousands of dollars over a long time. With MCMetrics, it only takes a few minutes to create and evaluate an A/B test.