gradient blur
Company

Confidence Intervals in Experiment Readouts

You can now see confidence intervals and a win percent for each variant within an experiment

A/B Testing

Jake Mor

Co-founder & CEO

Published

So you finally launched an experiment and are eagerly checking Superwall's dashboard for results ... we've all been there :)

Two questions will inevitabley come to mind:

  1. When can I consider an experiment finished?

  2. How much did our conversion rate increase?

These seemingly innocent questions actually require a ton of complicated math to answer. Until now, most customers resorted to using external AB testing calculators but we knew we could do better. Starting today, you'll see 2 new columns in an experiment's readout table:

gradient blur
dashboard-header

Win % and Confidence Interval (95% confidence) in experiment readouts

Here's what they tell you:

  1. Win % — This new column uses Bayesian inference to determine a winner. Unlike traditional methods that just give you a snapshot, Bayesian inference continuously updates the winning probability as new data comes in. This means you get a dynamic view of which variant is leading, incorporating all the data collected up to the current moment. This approach is more intuitive and reflects real-time insights into your experiment's performance.

  2. Confidence Intervals — On the other hand, the Confidence Interval column is grounded in the frequentist approach. It provides a range within which we are 95% confident the true conversion rate lies. This is crucial for understanding the reliability of your results. If the confidence intervals of two variants do not overlap, it's a strong indicator that you have a statistically significant difference. It's a more traditional approach but extremely powerful in determining the certainty of your results.

So how can we answer our first 2 questions?

When can I consider an experiment finished?

You know your experiment is complete when your Win % is greater than or equal to 95%. Superwall places a crown next to a variant when it gets there, but feel free to call it quits earlier if it makes good business sense. Remember – running more experiments should be your #1 priority.

How much did our conversion rate increase?

It's easy to forget how important this question is – especially when you are spending money on ads. Since a higher conversion rate changes the unit economics of your business, you need accurate data to scale ad-spend properly when a winner is declared. To be super conservative, consider your conversion rate might be at the lower bound of the 95% confidence interval. Ask yourself if you can stomach spend in this worst case scenario before scaling.

With these two powerful tools at your disposal, you'll have a clearer, more comprehensive understanding of your experiments. No more guesswork or external calculators; just solid, data-driven insights right at your fingertips.

Happy monetizing!

gradient blur

Get a demo

We'd love to show you Superwall

Want to learn more?

  1. Fill out this tiny form →
  2. We'll prepare you a custom demo
  3. Walk you through Superwall
  4. Follow up and answer questions

Key features

  • Drag 'n Drop Paywalls
  • 200+ Custom Templates
  • Unlimited A/B tests
  • Surveys, Charts & More
Select...

By proceeding you consent to receiving emails and our terms.

gradient blur
shape-starshape-starshape-starshape-starshape-star

Customer Stories

Our customers refer to Superwall as their most impactful monetization tool. In their own words:

dashboard-header

Thanks to Superwall, we were able to 2x our iOS app profitability in just six months. It has greatly assisted our growth team in achieving exceptional results by facilitating high-frequency experimentation.

Mojo launch
Bernard Bontemps, Head of Growth
dashboard-header

Really excited about the progress we made recently on paywalls with Superwall. We got more than 50% increase in conversion for upsell screens. This is crazy.

Photoroom launch
Matthieu Rouif, CEO
dashboard-header

Superwall has completely changed the game for us. We’re able to run experiments 10x faster and unlock the ideal monetization model for our users.

RapChat launch
Seth Miller, CEO
dashboard-header

Superwall made testing paywalls so much faster. Instead of releasing a new version of the app each time, we were able to iterate on the winning paywalls much quicker. Thanks to that it increased our revenue per customer by 40%.

Teleprompter launch
Mate Kovacs, Indie Dev
dashboard-header

Superwall lets us move 10x faster on our monetization strategy. We can build and launch multiple paywall iterations without the need for client releases or complicated deploys. Our product iteration loop is days, rather than months because of Superwall.

Citizen launch
Jon Rhome, Head of Product
dashboard-header

Superwall enables Bickster’s marketing team to design and optimize app paywalls, freeing up engineering to concentrate on innovation. As a result, Superwall helped accelerate our install-to-subscription rates, lower engineering expenses, and cured our team’s frustration with the (once) time-consuming process of iterating on paywalls.

Bickster launch
Chris Bick, CEO
dashboard-header

Superwall has revolutionized our monetization strategy. It’s an essential tool that allows rapid paywall testing and optimization, leading to remarkable improvements in our subscription conversions and revenue generation. Can’t recommend Superwall enough for any app-based business.

Coinstats launch
Vahe Baghdasaryan, Sr. Growth
dashboard-header

Superwall has played an integral part of improving our subscription business. Compared to other providers, Superwall has proven superior in facilitating high-frequency experimentation allowing us to achieve an ideal monetization model, resulting in a significant increase in revenue.

Hornet launch
Nils Breitmar, Head of Growth
dashboard-header

Superwall is the single greatest tool we’ve used to help us increase our revenue. Our material wins from Superwall are greater than any tool we’ve worked with to date!

Pixite launch
Jordan Gaphni, Head of Growth
dashboard-header

Shout out to Superwall for helping us dial in our paywall — made a big impact on monetization, increasing revenue by more than 50% 💸

Polycam launch
Chris Heinrich, CEO
dashboard-header

Superwall increases revenue. Full stop. Being able to test paywalls on the fly and quickly analyze results has drastically increased our revenue and improved our monetization of users. Highly recommend this tool!

Hashtag Expert launch
Zach Shakked, Founder
Start for FREE

Simple win-win pricing

Interest aligned pricing. Contact us for a discount.

dashboard-header
Indie
Free
Up to 250 conversions per month
Access to every standard feature
Try it free

Standard Features

  • 250 Conversions a Month
  • Drag 'n Drop Paywall Editor
  • 200+ Paywall Templates
  • Unlimited A/B tests
  • Charts & Analytics
dashboard-header
Startup
$0.20/conversion
Pay as you go pricing that scales
Up to 5,000 conversions a month
Sign Up

Standard Features

  • 5,000 Conversions a Month
  • Drag 'n Drop Paywall Editor
  • 200+ Paywall Templates
  • Unlimited A/B tests
  • Charts & Analytics
dashboard-header
Growth
Flat-Rate
100% custom flat-rate pricing
Terms that make sense for you
Get a quote

Premium Features

  • Unlimited Conversions
  • We Build Your Paywalls
  • 4 Weekly Growth Meetings
  • Dedicated Slack Channel
  • Custom Integrations