The Wikipedia fundraising team was performing up to 100 AB tests per week. It wasn’t enough to find the gains needed. The team needed to use statistics to interpret their AB tests accurately, but also to estimate smallest acceptable sample sizes to increase testing frequency. They were not comfortable with trusting methods proposed by other practitioners or academics who could not prove that their methods would work accurately for their data.<\/p>\n
The talk below presents simple methods that can accurately predict future performance from AB test results, and that allow you to determine the smallest acceptable sample size. Using four years of AB testing data, you’ll see how these methods really work. By taking a walk through four years of data, the presenters show how their current methods would have predicted the actual outcomes of long tests with much smaller samples, and the implications of various decisions about statistical confidence, power, sample size and choice of statistical method.<\/p>\n
The presentation will be useful to anyone who performs AB tests, manages teams that rely on AB testing, or holds responsibility for implementing or maintaining AB testing platforms.<\/p>\n
The presenters: Zack Exley, a practice director at the global IT consultancy ThoughtWorks. Previously, Zack was the Chief Community Officer and Chief Revenue Officer at the Wikimedia Foundation (Wikipedia), where he built an aggressive AB testing program that accomplished a 350% percent increase in revenue while radically reducing the length of the annual fundraiser. Also presenting is Sahar Massachi, a startup founder, online organizer, and software developer recently out of Brandeis with a BS and MA in Computer Science. After founding a start up to provide community organizations with better mobile tools, he joined up with the Wikipedia fundraising team to help complete a review of their AB testing methods, analyzing and preparing for publication hundreds of past AB tests.<\/p>\n
<\/p>\n