Features of an ideal A/B testing tool

The best experimentation software for most companies is a hub that covers most common use cases, with a strong API so you can plug point solutions into it. You can call this your core testing platform. It should include most of the following features:

It supports your platforms and programming languages

Does the platform accommodate your current and planned platforms? One tool that accommodates all your platforms will give you the best data quality. If everything is housed in one place, there’s less chance that discrepancies arise between systems, or that data gets stuck in silos.

Easy to install SDK (or no-install testing options)

The more work it is to set up the tool, the longer it’ll take and the more opportunities there are to make mistakes that’ll throw off reporting. Give preference to vendors that simplify the installation with an SDK.

Easy to use interface

The easier it is for non-technical team members to run tests, the more tests they can run and the more you can improve the product. Tools with unintuitive interfaces slow things and leave product and marketing teams reliant on IT.

Terms to look for: Code-Free Testing, Visual Editor, Visual Optimizer

Code-based testing

Even easy interfaces should still have an option for technical team members to run advanced tests. For example, e-commerce teams may need to use custom code to test big changes, like whether visitors buy more clothing when the online store displays two rather than three columns of products.

Bonus: Ask if there’s a code variable library. This lets developers save code-based experiments to run future tests quickly.

Flexible data export options

Do you have full control over your data? Any A/B testing tool should allow you to export raw testing data to a CSV file or other systems for analysis in other tools.

Breadth of integrations

Does it easily integrate with your existing tool stack? Look for lots of pre-built integrations to systems like Adobe, Google Analytics, Tableau, Amplitude, Looker, and more, plus a flexible API for the less common integrations.

Flexible reporting and event tracking

Can you customize the reports and define your own custom events? If not, it had better have a good integration to whatever analytics tool you plan to use.

Powerful segmentation

How deep does the segmentation go? Can you filter by multiple variables? Custom variables? Does the system support random bucketing, saved segments, and cohort tracking?

Server- or client-side testing options

Does the tool allow you to define whether tests are run server-side or client-side? Depending on your site or app, you may need both.

High number of simultaneous users or experiments

Does the tool support your desired number of users and experiments on the plan you’ve selected? Will your experimentation software remain affordable as you scale?

Flawless test delivery

Look for a system that runs imperceptible tests. When a tool is built poorly and is slow at delivering A/B tests, users sometimes see a flicker effect, which degrades their experience and either makes the app look like its malfunctioning or alerts users that they’re part of a test.

Mobile testing that doesn’t require an App Store update

The two big app stores require documentation and approvals for app updates that can slow teams down. Look for an A/B testing tool that allows mobile testing without App Store updates.