Calculating Marketing ROI: If You Can’t Measure It, Don’t Invest

Having trouble calculating marketing ROI for your marketing campaign? My advice is to pull the plug.

In my days as an equity analyst, I would occasionally be asked to give my opinion on a financial corporation that my portfolio manager was considering as an investment.
Almost every time, my answer was, ‘don’t invest.’
I didn’t take that position because I believed it was over-valued, but because I simply had (and still have) no idea what the vast majority of financial companies are worth. For an investment bank with 20x leverage, the difference between being insolvent and compliant is a change of just 5% in the value of its assets. A clever and adequately unprincipled accountant can easily — and legally — fudge assets by 5% using inflated carrying values on illiquid assets. If I can’t tell you whether the company is or isn’t insolvent, I definitely can’t tell you if it’s worth more or less than $16.45.
Instead of giving an answer in which I had virtually zero confidence, I abstained on the financial sector altogether, saving my brainpower for all the companies out there that are transparent enough to analyze.
I feel the exact same way about marketing.
Marketers have all sorts of options for how to promote their product, both online and in the tangible world, and each and every one of these options has the potential to drive sales. But a marketer’s job isn’t just to drive sales. They’re also charged with:

1. Generating a positive ROI from their marketing efforts
2. Increasing their ROI over time by optimizing their spend

Without the possibility of calculating marketing ROI with some precision, a marketing channel to me is just like a financial stock. You know it’s not worthless, but you really have no idea how much it is worth. And if you can’t measure the value it brings to your company, then you can’t calculate an ROI or compare it to other campaigns. You’re better off not playing.
A perfect example is billboard advertising.
I’ve long thought of billboards as the single hardest ROI to calculate in the marketing world, so I found a SlideShare presentation that makes a valiant attempt. I’ve excerpted the page where the author calculates the ROI of his/her Honda billboard campaign here:

Seems pretty reasonable. But there’s a big assumption here, which is that the billboard was responsible for 8 car deals. How did he/she come up with that number?
There’s another slide for that:

Here’s where it gets messy, for three reasons:

1) Survey tactics matter

The ROI is basically derived from a survey question, and how the survey phrased that question matters A LOT.
For example, asking the question, “Did seeing our billboard influence your decision? (y/n)” will get a much higher positive response rate than a long picklist of sources, especially if the “billboard” entry is buried in the list. Entrants who saw the car in multiple places before purchasing it may just pick the one they come across first. Suffice it to say that number could easily have been either 2 or 4 if the tactics were different. It’s not a simple binary fact.

2) Tough sample size

When the number of positive responses is this low, the confidence interval is not kind. At a 95% confidence level, the number of actual responses out of the 189 total buyers is somewhere between 3 and 13. Not all that precise.

3) Intangibles to Consider

The fine print at the bottom of the slide hints that there is hidden, intangible value in the campaign from partial credit and brand perception/awareness. That’s good for the value of the campaign, but how good? We have no way of knowing.

Plus, of course, the intangible benefits, which we have no way of measuring. 
So, the campaign delivered somewhere between zero and a 6x return, or in other words, it either barely broke even or was outrageously successful.
In a vacuum, I’d probably pull the trigger on this campaign, because there’s a low likelihood that it actually loses money. But if I’m also asked to optimize my marketing ROI, these results are not nearly conclusive enough to compare against other options and therefore don’t allow me to do my job.
I’m not writing this to pick on this author (who I think did the best he/she could), or even billboard advertising in general. Technology is not a silver bullet, and I actually think many online marketing channels face a similar problem.
If you’re a B2B company with a long-cycle sales process, a prospect may be evaluating your product for months on the internet, asking their peers about you, and interacting with salespeople, so it’s not always obvious which campaign was responsible for bringing them to your website or converting them into a customer. Even if it is technically possible to do so, many companies lack the time or expertise to properly attribute leads to the right channel.
My advice is, if you can’t measure it, don’t invest in it. You don’t have to take advantage of every marketing channel, so stick to the ones you can actually analyze, even if that means sitting out on some logical-sounding opportunities.

Do you agree you should never invest in a marketing channel you can’t accurately measure?