Here at Engage, we A/B test websites, subject lines, copy — pretty much anything and everything — without a second thought. But surprisingly, A/B testing in mobile apps is much, much more difficult and is years behind A/B testing tools for the web.
However, there are a couple solutions out there if you want to A/B test your iOS app. Unfortunately, many of them require you to write tests into your code or create multiple storyboards and then resubmit to the App Store (like Clutch.io and Amazon’s still-in-beta mobile A/B testing service) or mess with various tools to perform just a simple test (like pathmapp.com). But these methods seemed inefficient and onerous — having to re-submit your app to the App Store every time you want to run a test means you not only have to wait for approval from Apple, but for your users to update the app as well. With such a long lag time and inability to tweak tests while they’re running, many of the benefits of A/B testing are lost. And it seemed ridiculous that you should have to paw through your code every time you want to run a test.
Thus, we thought there must be a better way to do things. So, we decided to use this past weekend’s AngelHack hackathon to try out a new approach. AngelHack is a worldwide series of hackathons that is being held in 35 cities this year and culminates in a final competition in San Francisco. This was DC’s second year as a host city, with the event held at the 1776 incubator space on 15th Street. I was joined by Engage’s Michele Hansen (who served as Product Manager/Idea Person) and Will Summers, a Graphic Designer at Interface Media Group, who we met at the hackathon.
We had 24 hours to build. Here is what we learned.
We started out by defining our requirements: it should be easy to implement into an existing iOS app without re-submitting to the App Store once the SDK is added and it should be possible to create tests on the fly – even for non tech-savvy people.
This led us to building a web interface inspired by Optimizely and Unbounce. We call our framework FlipTest.
The FlipTest framework builds a full overview of the app’s different screens (ViewControllers) and view hierarchies and then sends this information to the FlipTest.io website via a simple REST API. This allows the website to visualize the different ViewControllers and provides an easy way for users to A/B test changes to elements such as labels and buttons.
The view-hierarchy generation (to be sent to the server) and the view-injection process (making changes based on the A/B testing profile) is implemented using method swizzling. This allows us to very lightly inject into the existing code base of the app at runtime and perform view updates and traversal in a very efficient way. And since the code isn’t executable, it doesn’t violate App Store policies.
The FlipTest integrated app will then periodically ask for a testing profile via the REST API and make any changes to the UI if necessary. This also enables the app to track unique views and clicks so a conversion rate can be calculated and visualized on the website.
We currently show simple conversion rates visualized on a line chart. We’re looking to improve this along the way by adding more intelligent feedback for the collected data.
For a quick overview of what all this looks like, check this awesome video that @willsummers made just before we handed in the project:
We didn’t end up winning any prizes, but considering we went into the weekend without knowing if what we wanted to do was technologically possible (nevermind something we could build in a weekend), we’re incredibly proud of what we built and hope to release a version of FlipTest in the near future. You can sign up for updates about FlipTest here.
Engage’s Mathias Hansen (Lead Developer) and Michele Hansen (Operations Manager) pitching FlipTest at AngelHack DC. Photo taken by Evan Burfield
So what do you think about this approach? And is this something you would like to be able to use in your app? We would love to hear some feedback and comments.
FlipTest.io was built by @MicheleWalk, @MathiasHansen and @willsummers at AngelHack DC. (Cross-posted from FlipTest.io)
The post How we built an A/B testing framework for mobile apps in 24 hours at AngelHack DC appeared first on Engage.
