Are your app installs incremental? Measuring with imperfect data

Part of your paid app installs would have happened without ads too, and that is cannibalisation of organic. To know what your ads truly add, you measure incrementality with a simple holdout test instead of trusting platform numbers blindly. Here is how to decide well, even with imperfect data.

Not every paid app install is incremental. Part of the people who saw your ad and installed your app would have done so anyway, even without the ad. Your platform claims those installs as a result, while they would have happened organically. That is called cannibalisation, and it is why your platform numbers often look better than your total growth justifies. To know what your ads truly add, you measure incrementality with a simple holdout test. This article explains how cannibalisation works, how you set up a holdout and how you decide well with data that is never perfect.

What does incrementality mean for apps?

Incrementality is the number of installs that would not have happened without your ads. That is a different number from the installs your platform attributes to your campaign. The platform counts every install preceded by an ad contact, even if that person already wanted your app. For apps this is extra tricky, because people also find your app through the store, through word of mouth or through your brand. So the question that really matters is not how many installs your campaign claims, but how many installs disappear if you turn the campaign off. That difference is your true impact.

How do you spot cannibalisation of organic?

The signal of cannibalisation is a mismatch between your platform numbers and your total growth. Your ad platform reports a thousand installs this month, but your total installs rose by only four hundred. Where did the other six hundred go. Often they sat in your organic stream, and your ads only pulled them into the paid channel without enlarging the total growth. This is not proof that your ads do not work, but it is a warning that your platform numbers overstate the truth. Whoever scales blindly on those numbers pays for installs they would have gotten anyway.

How do you set up a simple holdout test?

A holdout is the most honest way to measure incrementality, and it does not have to be complicated. The idea is that you deliberately show no ads to part of your potential audience and compare how many installs that group produces anyway. That can be through a geographic holdout, where you turn your ads off in a comparable market or region and compare the organic installs against a region where you do advertise. The difference between the two is an estimate of what your ads add. The important thing is that the two groups are comparable and that you measure long enough to average out chance.

• Choose two comparable markets or regions and turn your ads off in one while you keep advertising in the other.
• Measure the organic installs in both and compare the difference, because that is your rough incrementality.
• Let the test run long enough so daily fluctuations and seasonal noise average out.
• Repeat the test occasionally, because cannibalisation changes as your brand becomes better known.

Why do you decide better with imperfect data?

Many teams postpone incrementality because they fear a holdout is not clean enough. But the alternative, blind trust in attribution, is not more neutral, it is systematically too optimistic. A rough estimate of your true impact is more usable than a precise number that steers you in the wrong direction. You do not need to build a laboratory. You only need to know whether your ads truly raise total installs or mainly move organic installs. Even an imperfect holdout gives you that direction, and with that direction you make better budget decisions than with a dashboard that tells you what you want to hear.

A rough estimate of your true impact is more usable than a precise number that steers you wrong.

What do you do with the outcome?

You translate the outcome of your holdout into your decisions, not into a spreadsheet that disappears in a drawer. If your incrementality turns out high, you know that scaling produces real growth and you can add budget with confidence. If it turns out low, you are mainly paying for installs you would have gotten anyway, and the honest conclusion is that you need to change your approach or targeting before you spend more. In both cases you steer on the true impact instead of the claimed impact. That way you avoid scaling for months on a number that never reflected your total growth.

Conclusion

Not every paid app install is incremental, because part of it would have happened organically. Cannibalisation explains why your platform numbers look better than your total growth justifies, and a simple holdout test shows you what your ads truly add. You do not need to measure that perfectly to decide better with it. Want to tackle measuring the real impact of your app installs and steer your budget on growth instead of on claimed installs? Book a call and we will gladly look at it with you.