From the Review of the U. of Chicago Booth school of business:
Why the power of TV advertising has been overstated
BRIAN WALLHEIMER | JUN 18, 2019
Television advertising may be considerably less effective than published studies suggest, according to Chicago Booth’s Bradley Shapiro and Günter J. Hitsch and Northwestern’s Anna E. Tuchman. Their findings may disappoint marketing executives responsible for spending billions of dollars a year on TV ads in the United States.
Drawing on the Nielsen Datasets at the Kilts Center for Marketing, the researchers devised a way to match up data on advertising with data on sales for the top 500 products sold at more than 12,000 stores from 2010 to 2014. The results demonstrate that television advertising’s effect on sales over the 52 weeks following the ads was about one-fifth as much as calculated when considering only data that showed statistically significant positive results—the only type of results typically included in published studies of advertising efficacy. A typical brand in the data sample, if doubling its television advertising, should expect about a 1 percent increase in sales, as compared to the 5 percent it might expect if it analyzed only successful campaigns, the research suggests. …
Shapiro, Hitsch, and Tuchman argue that while most brands in their sample would benefit from advertising less, many are still better off with their current advertising expenditure than if they weren’t advertising at all. However, for almost all brands, shutting off some advertising would increase profit.
So why has the established literature found larger average ad effects than Shapiro, Hitsch, and Tuchman find? The researchers suggest publication bias may be a big part of the answer. Academic journals are interested in studies that show statistically significant results, specifically increases in sales or returns on investment related to advertising. So when researchers find little or no effect, it’s likely that journal reviewers will reject their studies. “In particular, editors or reviewers may reject advertising-effect estimates that are not statistically significant or judged as small or ‘implausible,’ i.e. negative. Thus, false positives get published while true negatives get discarded,” the researchers write.
And that’s if the researchers even try to publish in the first place. The field has a documented “file drawer problem,” Shapiro, Hitsch, and Tuchman say, wherein research sometimes goes unfinished or unsubmitted to journals if the researchers recognize the findings are unlikely to be published. Including all the data, they argue, reveals that TV ads have a much smaller effect on sales than assumed. The researchers find that when they restricted their estimates to only brands that saw a positive and significant effect of advertising on sales, the average advertising effectiveness was much closer to that which is estimated in the established literature.
This is exactly what my experience was when I was in the test marketing business in Chicago in the 1980s.
I worked for a start-up that had constructed the world’s all-time best test marketing real world laboratory. We had eight small cities around the country like Eau Claire, Wisconsin in which we had paid to install in all the supermarkets in town the new laser-beam checkout scanners. In return, the supermarkets cooperated with us by scanning the ID cards of the 3,000 households we recruited in each town to identify themselves each time they went through the checkout line. So we had the complete shopping history of 3,000 families in each city. Each household was a cable TV subscriber, and we had installed a converter box on top of their TVs that allowed us to manipulate what TV commercials they saw: e.g., we could choose whether they saw Bill Cosby endorsing Jello Pudding Pops or whether they saw a public service announcement about how only you could prevent forest fires.
This system that opened at the end of 1979 probably remains the greatest real world experimental laboratory in the history of the social sciences.
From 1980 through 1985 it was immensely popular with the brand managers of the top CPG firms in the country. They tended to be convinced that they had overseen the creation of such hugely persuasive advertising that the only thing standing in the way of higher sales was top management’s mule-headed refusal to double their advertising budgets. So we got hired to do hundreds of year-long test markets for six figures each in which we showed half the people in town the current national advertising level and half double the current level. We made sure to make the test group and the control group exactly equal in purchases of the test brand and the test category over the previous year. This was SCIENCE!
And … because what we did was scientific, in the great majority of cases it turned out that doubling the advertising didn’t increase sales by a statistically significant amount.
I did manage one test of a famous toothpaste brand whose scientists had actually invented a new ingredient that was so effective that the American Dental Association endorsed their upgrade as something that everybody should start using immediately. In that case, it turned out that extra advertising about how the ADA wanted them to buy this new version NOW actually did boost sales.
But in the usual case of a famous old brand just repeating its usual catchphrase more often, higher advertising didn’t boost sales more than our quite sensitive level of statistical significance. This studies overall result that doubling advertising increased sales by on average by 1% sounds very much in line with my experience.
One time I tested both boosting and cutting ads for an extremely famous brand that you may well have in your bathroom. Neither moved the needle at all. I recommended to the client that they should consider testing advertising cuts for all their brands.
My strategy was forcibly rejected. It was explained to me that brand managers at this firm didn’t get promoted for cutting advertising they got promoted for convincing top management to boost their advertising budgets because they were such geniuses at advertising. This firm believed in advertising. After all, the term “soap opera” had been invented to describe the early TV shows sponsored by this firm.
An interesting question is whether online advertising is so much more effective than TV advertising. If it turns out not to be, how much does the NASDAQ average drop the next morning?