The growing range of advertising effectiveness measurement methodologies:Characteristics and best ways to choose

Advertising communication planning is becoming increasingly complex with the diversification of media, and new outlooks, methodologies, and tools to measure effectiveness being developed in line with this.
At the same time, many companies are struggling with what to base their choice of effectiveness measurement methodologies on to suit their company’s issues from the diverse options available. In addition, if the design used to measure effectiveness is not properly put together, it may not even deliver correct evaluations.
In this article, we will summarize the current state of advertising effectiveness measurement, as well as delineate how to choose a methodology to suit your objectives, and how effectiveness measurement methodologies are designed.

*This article is a reconfiguration of the content of our “Cross-Media Advertising Effectiveness Measurement: Methodologies and Practice” seminar conducted on November 28, 2023.

The need for advertising effectiveness measurement

The environment surrounding advertising has changed significantly in recent years due to the diversification of media. Let’s first look at the changes in consumer media exposure behavior. Diagram 1 presents the trends in usage rates by device along with TV and smartphone usage rates by age group, tabulated from a device usage survey implemented by INTAGE.
TV (terrestrial) still displays a higher usage rate than smartphones, but is declining year by year. This validates the impression that young people are turning away from TV, with the decline in usage rates particularly marked among those in their 20s and 30s. Smartphone usage rates are conversely dramatically up. This increase is particularly distinct among those in their 40s ~ 60s, and there seems to be room for further growth in future as well.

Fig 1

In addition, on observation of the shifts in advertising expenditure by media type in Fig 2, corporate expenditure on online ads has increased sharply, exceeding TV media advertising expenditure in 2019.

Fig 2

As observed here, consumer media exposure behavior is constantly changing, with the media in which companies invest also changing significantly. The cost-effectiveness of marketing budgets is also being brought into question more than ever before, making advertising effectiveness measurement all the more important.

What makes advertising effectiveness measurement hard?

Clients often ask questions about advertising effectiveness measurement like “Can you visualize the conversion effect on a per-person basis (single source) such as the increase in purchases due to ad exposure?”, “I’d like to visualize the effect across media; what sorts of methodologies are available?”, “Can’t the effects of TV commercials and digital ads be evaluated utilizing log data?” and “I’d like to implement MMM (Marketing Mix Modeling). What sorts of methodologies are available?”, indicating a tendency to focus on methodologies.
The reason for their focus on methodologies is that the environment is changing rapidly, including the digital shift, technological advancements, and the protection of personal information, making it hard to keep up with information.

We have summarized the changes in advertising effectiveness measurement methodologies below in Fig 3. Up until 2012, advertising deployment centered on TV, so evaluations of TV commercial creatives via questionnaire were the mainstream. From 2013 onwards, the digital shift has accelerated, with a large number of questionnaire methodologies using digital ad exposure logs being released.

Fig 3

Since 2018, the handling of personal information has become stricture due to GDPR coming into force, placing restrictions on surveys using ad exposure logs, so MMM (Marketing Mix Modeling) that enables effectiveness to be measured without handling personal information are once again currently attracting attention. In addition, while individual-base evaluation methods are evolving such as INTAGE now being able to measure YouTube effectiveness based on ad exposure logs, surveys with ad exposure logs that were once possible are not any more.
At present, there is no “all-purpose methodology” for advertising effectiveness measurement, so an advertising effectiveness measurement methodology needs to be chosen with the outlook the means to obtain this with will differ depending on the purpose of use.

Choosing a methodology to suit your objectives

So, what is the best way to think about choosing a methodology to suit your objectives? The process for this is depicted in Fig 4. The main purposes of effectiveness measurement can be broadly divided into three: “appropriate allocation of marketing budget” on a high business/brand level, “integrated ad/campaign PDCA management” on a campaign level with multiple media deployment, and “improved ad effectiveness/efficacy” on an individual media level.

Fig 4

If the purpose is “appropriate allocation of marketing budget”, we choose MMM (Marketing Mix Modeling). MMM is a methodology that statistically analyzes shifts in the impact of marketing investment on performance such as sales chronologically. It can clarify each marketing measure’s contribution/ROI on sales, and calculate the appropriate allocation for each measure. Conversely, it isn’t a methodology that can inform consideration of detailed actions since it doesn’t analyze individual-base evaluations like awareness, overall liking, image, or purchase intention.

If the purpose is “integrated ad/campaign PDCA management”, 3 methodologies are available.
Firstly, in cases where we are fine with identifying changes in awareness, image, or purchase intention, we choose a “Simple KPI questionnaire survey”. Conversely, in cases where we need to identify precise effects across media, we choose an “ad exposure opportunity estimate (*OTS)-based questionnaire survey” or a “multiple media ad exposure log-based questionnaire survey”. If exposure to media deployed in a campaign can all be obtained via log data, we recommend choosing an ad exposure log-based survey; however, since this often can’t be obtained at present, we recommend an “ad exposure opportunity estimate (OTS)-based questionnaire survey”.

If the purpose is “improving the effectiveness/efficacy for each media”, we choose an “individual media ad exposure log-based questionnaire survey” or “exposure efficacy data (such as CPM or CPV)”, and analyze the effect on each media in depth in terms of frequency and ad menus.

In this way, the methodology differs depending on the purpose of use and level of validation required. It is also important to choose an appropriate advertising effectiveness measurement methodology with cost constraints etc. in mind as well.

A representative cross-media research methodology

Lastly, let’s take a slightly more in-depth look at the differences between the three methodologies used for “integrated ad/campaign PDCA management” purposes, which we receive the most requests for recently. We have summarized these in Fig 5 along two axes for media coverage and appropriacy of result.

Fig 5

“Media ad exposure log-based questionnaires” target people who have definitely been exposed to ads, so their media evaluations are highly accurate and persuasive. However, since ad exposure logs can only be obtained from some video services and social media, its media coverage is limited. From a personal information perspective, it is expected that it will remain difficult to capture digital ad exposure across media via logs in future.

On the other hand, “simple KPI questionnaires” that do not use ad exposure logs and survey people who respond they “have seen the ad” offer broad media coverage and are relatively low cost, so are easily adopted, although bias in their response results need to be kept in mind. As depicted in Fig 6, “people aware of ads” tend to include “people highly involved in the category” and “people highly involved with the brand”, meaning they may deliver exaggerated evaluations of the ad’s effects. While highly versatile, their demerit is that they can be highly biased, with evaluations less appropriate.

Fig 6

Both these research methodologies do have issues, with “ad exposure opportunity estimate (OTS)-based questionnaires” conversely relatively balanced. As depicted in Diagram 7, they are a methodology that validates consumer ad exposure opportunities – that is, reach – based on “consumer mediate activity and habits” and “state of ad deployment in each media”. While ad exposure opportunity estimate (OTS)-based questionnaire surveys offer an estimate of exposure to digital ads, they offer both coverage and appropriacy and are a promising methodology in terms of both versatility and accuracy.

Fig 7

This approach delivers a cross-media visualization of reach, ad awareness wins, and effects on attitudinal change as seen in Fig 8 and Fig 9. It also clarifies whether the ads effectively win in terms of goal indicators based on the relationship between spending on ad deployment and its effect (number of people won over in terms of awareness, purchase intention, imagery etc.), and conducts an evaluation with “room for growth” taken into account.
The results of these validations can then be used to improve your next wave’s media plan (media selection and allocation).

Fig 8

Fig 9

Conclusion

At present, there is no “all-purpose survey methodology” that can handle everything, so a methodology needs to be chosen with its suitability in mind, since the ways forward differ depending on one’s objectives.
Consumer media activity is evolving day by day, so we predict that what advertising effectiveness measurement can and cannot do will change accordingly. In future, it is likely that it will be important to continue to keep up with the latest information on individual advertising effectiveness measurement solutions.

At INTAGE, we leverage unique data and know-how in our advertising effectiveness measurement introduced in this column to provide support for advertising PDCA in line with your issues and state of ad deployment. Should you have any questions or concerns, please feel free to contact us.

Reproduction and Quotation

◆This report is copyrighted by INTAGE Inc.
Please check the following prohibitions and precautions, and specify the source when reproducing or quoting this report.
“Source: INTAGE “Knowledge Gallery” article published MM/DD/YYYY article”

◆The following are prohibited:
・Alteration of this article in full or in part
・Sale or publication of this article in full or in part
・Uses that are against public order and morality, and uses connected with illegal activities
・Reproducing or quoting panel data* for the purpose of advertising or promoting companies, products, or services
*Panel data include: SRI+, SCI, SLI, Kitchen Diary, Car-kit, MAT-kit, Media Gauge, i-SSP, etc.

◆Other precautions:
・INTAGE Inc. shall not be liable for any trouble, loss, or damage caused by the use of this report
・These usage rules do not restrict the use of quotations or other uses permitted under the Copyright Act of Japan

◆For inquiries about reproduction and quotation, click here