Causal Impact

Causal Impact is a statistical method for measuring the incremental effect of Google Ads without a traditional control group.

Not every advertising measure can be accompanied by a test. Often the campaign has already run, and the question about its effect comes afterward. Causal Impact is one way to estimate it from time series.

Where the term comes from

The method comes from a paper by Kay Brodersen and four colleagues, published in 2015 in the Annals of Applied Statistics. The authors worked at Google. The paper examines the effect of a planned market intervention, such as an advertising campaign, on a metric over time.

As an example, the paper uses an online campaign and its effect on visits coming from search. The method is implemented in the R package CausalImpact, which Google provides as open source.

How Causal Impact works

The package builds a Bayesian structural time-series model. In a pre-period it learns how your target metric behaves relative to comparison series. From that it calculates how the metric would have developed after the measure without it. This is called a counterfactual scenario.

The difference between the forecast and the actual values is the estimated effect. Results can be output as a table, as a description in words or as a plot.

The documentation names clear prerequisites. You need one or more control series that were not affected by the measure. If this assumption is violated, the effect may be wrongly estimated or falsely detected. In addition, the relationship between the control series and the target metric must remain stable in the period after the measure.

How it differs from experiments and lift tests

A holdback experiment deliberately keeps part of the audience away from the advertising and compares the groups. That is a true experiment. Causal Impact needs no split, but it calculates after the fact and relies on assumptions.

A conversion lift test also measures the effect using test and control groups that are set up before launch. That usually gives the more reliable statement, but it requires the test to be planned in advance.

The umbrella term is incrementality testing: the question of what an advertising measure achieves on top of what would have happened anyway. Causal Impact is one method within that question, not a replacement for an experiment. The authors themselves discuss strengths and limitations for the case where a randomized experiment is not possible.

What this means for your advertising

Think about the evaluation before you start. Record when the measure began, and secure enough data from the period before. The pre-period should contain the fluctuations of your business, such as weekdays and seasonal patterns.

Look for suitable comparison series. These can be regions where you changed nothing, or channels that are not affected. Check that these series really do not feel the measure. A brand campaign that also influences organic search makes organic clicks a poor comparison series.

Typical questions are the effect of a new campaign or a budget increase. The result is an estimate with a range of uncertainty, not proof. A report should show both.

Limits and typical mistakes

The biggest mistake is a comparison series that is itself affected. The model then delivers a clean-looking but wrong result. The documentation warns about this explicitly.

Second, several things often change at once, such as price, promotion and ad budget. The model then cannot attribute the effect to a single cause.

Third, the method needs data that is long and stable enough. With short series or strong breaks, the forecast becomes uncertain. A planned test is then the better choice.

Frequently asked questions

Do I need no control group for Causal Impact?

No group of users who deliberately see no advertising. You do need comparison time series that were not affected by the measure, such as data from uninvolved regions or other channels. Without such series the estimate is not reliable.

Is there a Python version of Causal Impact?

Google’s original is an R package. For Python, the package’s readme recommends the implementation “TFP CausalImpact”, which is based on TensorFlow Probability and is designed to produce similar results.

Is a simple before-and-after comparison enough?

No. A simple comparison overlooks trend, seasonality and other influences. Causal Impact models these through comparison series and estimates how the figure would have developed without the measure.