20bits - Latest Comments in Statistical Analysis and A/B Testing

Re: Statistical Analysis and A/B Testing

jtregister — Wed, 26 Aug 2009 13:19:14 -0000

This is a great resource; thanks. I'm beginning to look into the statistics behind A/B testing and have some questions. This is well after the initial post, so hopefully Jesse and others will see this.

For the distribution of the conversion rate, it seems like it should be a binomial distribution, which can be approximated by the normal distribution (as Jesse asserts in the comments) with scale.

But how about if we take this one step further and look to measure this on an e-commerce website, where there's not just conversion rate but also average order value to consider? (Really, we want to look at the contribution margin, but let's assume -- admittedly incorrectly -- that we have a 100% margin on the shopping cart.) This considers contribution per visitor, a broader metric of an e-commerce website than simply conversion rate. (And of course the subsequent step is to follow the impact on lifetime customer value, but let's not go there for now.)

Now if you consider the distribution of average order value on a typical e-commerce website, often ~95% do not convert. Of those who do convert, there's typically a normally distributed range of average order values. But if you plot the entire range of AOV, including those who don't convert, there's a huge 'peak' at zero followed by a normal bell curve. This is a more complicated distribution than a simple normal distribution.

Does anyone have insights on how to analyze the A/B results for contribution per visitor given this type of distribution? Seems like perhaps a compound Poisson, or something similarly complex. Or can someone perhaps provide a good justification of why this level of complexity is unnecessary in the analysis?

Thanks,
Jonathan

Re: Statistical Analysis and A/B Testing

Loïc d'Anterroches — Tue, 25 Aug 2009 08:06:37 -0000

Thanks a lot! You helped me getting a 50% boost in conversion rate for my website:
http://www.ceondo.com/ecte/2009/08/ab-testing-b...

I really recommend everybody to do some AB testing. I am linking to the PHP code to the tests from my article if people are interested.

Re: Statistical Analysis and A/B Testing

Traveller_Adventure — Sun, 02 Aug 2009 00:28:05 -0000

This is quite impressive, I am pleased to read this post, keep posts like this coming, you totally rock!
Cheers,
Blog Review

Re: Statistical Analysis and A/B Testing

Yvonne — Mon, 19 Jan 2009 10:19:09 -0000

If for example, you had a treatment D which had a z-score of -2.94 - would you then be 95% confident that treatment D is worse than the control?

Re: Statistical Analysis and A/B Testing

jfarmer — Sun, 16 Nov 2008 12:55:48 -0000

"The conversion rate for each treatment approximates a normally distributed random variable" is more correct.

Re: Statistical Analysis and A/B Testing

hadley — Sun, 16 Nov 2008 10:40:34 -0000

"The conversion rate for each treatment is a normally distributed random variable" - are you sure??

Re: Statistical Analysis and A/B Testing

hadley — Sun, 16 Nov 2008 10:39:32 -0000

If you're going to use R, why not actually use the appropriate test - in this case it would be prop.test() for testing the different between two proportions.

Re: Statistical Analysis and A/B Testing

jfarmer — Sat, 15 Nov 2008 16:24:27 -0000

Oh, that was weird.

Re: Statistical Analysis and A/B Testing

jfarmer — Sat, 15 Nov 2008 16:24:16 -0000

Where did my comments go? :(

Re: Statistical Analysis and A/B Testing

Brendan O'Connor — Thu, 13 Nov 2008 18:33:39 -0000

The most important thing to know is a software package to use -- you don't want to muck around coding this yourself. R's t.test() is a good choice. (I guess Excel is good too.)

Re: Statistical Analysis and A/B Testing

Brendan O'Connor — Thu, 13 Nov 2008 18:33:21 -0000

The most important thing to know is a software package to use -- you don't want to muck around coding this yourself. R's t.test() is a good choice. (I've heard Excel can do it too I suppose.)

Re: Statistical Analysis and A/B Testing

Chris — Thu, 13 Nov 2008 01:21:20 -0000

They were using an Excel spreadsheet and were kind enough to make it public.
http://blog.socialmedia.com/wp-content/uploads/...

I just tried it and had to watch the video a couple of times to understand how to use the spreadsheet.

Re: Statistical Analysis and A/B Testing

Jesse — Thu, 13 Nov 2008 00:36:05 -0000

Chris,

That's a nice video. I wonder what software they were using?

Re: Statistical Analysis and A/B Testing

Chris — Thu, 13 Nov 2008 00:29:23 -0000

Nice post. Social Media had a similar blog entry a while back.
http://blog.socialmedia.com/crafting-a-statisti...