I'm definitely the first to suggest A/B testing. It's worthwhile to extol it's numerous uses, but it's no silver bullet. A claim like this: "the best ad network is the one that, for a given level of traffic, offers the highest RPM." is itself a guess. Data can help drive to a decision but it doesn't make the decision. For instance, what about offensive content in the ads? It might drive your RPM up but your total pageviews down. Or it might prevent you from reaching a new audience in three months when you try to enter a more genteel market, something which your initial A/B test won't illuminate.
The key is to make data driven decisions, which means balancing analysis, synthesis and data.
Jesse
· 1 year ago
Timothy,
I should've added a footnote, but I agree. You can basically make your measurements as complex as you want, but there's some point of diminishing returns.
You could, for example, measure the effects of the ads on the growth of your userbase and then make some sort of "net present value" calculation.
Really all you need is enough data to make an informed decision — some data is better than none.
All Comments
The key is to make data driven decisions, which means balancing analysis, synthesis and data.
I should've added a footnote, but I agree. You can basically make your measurements as complex as you want, but there's some point of diminishing returns.
You could, for example, measure the effects of the ads on the growth of your userbase and then make some sort of "net present value" calculation.
Really all you need is enough data to make an informed decision — some data is better than none.