Chapter 7: Applying indicators – a campaign planning and evaluation approach

This chapter outlines frameworks for incorporating influence indicators into structured, system-based approaches to analysis and decision-making.

Two general frameworks are proposed. One facilitates assessment of populations (from individuals, communities, and societies to nations), using combined indicators to undertake what is variously called situational understanding, market research, or audience analysis. A second approach uses the stages of an information influence campaign, from goal setting through planning, implementation and monitoring, to evaluation – an approach common in public relations and strategic communication. These frameworks can used in tandem: campaign planning typically (and ideally) includes audience analysis.

These frameworks can be applied to positive, attributed campaigns of public persuasion and diplomatic engagement. They can also be applied to analyses of threats posed by malign influence campaigns, including those covertly undertaken by foreign actors. The approaches are similar in both cases although there is much less agreement on the standard or best practice approach to the analysis of malign influence. (This is discussed in Chapter 5.)

This chapter has three sections:

1. An outline of the elements that are included in audience analysis and campaign planning;

2. A framework for campaign planning and evaluation, based on best practice; and

3. Examples of approaches to analysis of malign influence campaigns, based on emerging research.

Elements for integrated assessment of influence indicators

The main elements of a campaign assessment are: actors; activities; impacts and effects; and contexts.

Actors

Actors include proponents (or ‘authors’) and target audiences, noting that in most assessments of an information environment, actors can be both authors and audiences, creating and (re)distributing message content via communication networks (which they are a part of). Actor analysis incorporates indicators that identifies actors, describes their characteristics, and seeks to discern their strategies (their campaign goals and their target audiences).

Assessment of actors asks the following questions:

1. How many actors can be identified? Are they acting in unison, in a coordinated manner, independently, or in competition with each other?
2. Can actors be identified? Are they recognised and legitimate? Are they operating covertly or disguising themselves? Are they a (foreign) state actor or state-aligned, or a non-state actor?
3. What is the organisational structure of the actor/s? Is it an individual, a small group, an organised network, an institutionally structured organisation? What is the leadership and administrative structure?
4. What are the assets and capabilities: personnel, budget, expertise and the like?
5. What is the ideology of the group, and its purpose (what are the security, political, or social objectives)? Is the goal the maintenance or legitimisation of power? Is the aim to arouse people in the promotion of a cause?20
6. What is the strategy (see below)?

All these questions will be relevant when assessing influence operations conducted by another actor; when planning one’s own influence activities, questions 3-5 are pertinent in the initial stages of preparation and pre-planning).

Strategy and tactics

Strategies and tactics utilised by influence actors can be identified and categorised according to the approach and goal of the influence activity. Critical elements of the strategy include (1) the identity and characterisation of the ‘messenger’ and the (2) targeting of the audience. These are typically undertaken prior to the communication of campaign content, although may not be evident until the content appears and can be analysed.

The identity of the messenger is an essential indicator of influence strategy. Identities can be determined by identification of the sources of messages in the information environment.

The messenger may be open, attributed and branded, or covert, fake and/or unbranded. It may be represented as a credible authority, a like-minded community member, among others. In open, attributed campaigns, the identity may be determined easily; in covert or limited attribution campaigns, identifying of the source of a message may be require tracking and tracing narratives through various channels.

Identities can include public personas and profiles that, even when legitimate, are always constructed for two purposes. First, appearing to share the identity, values and motivations with the intended targets increases the likelihood messengers will be trusted and messages will be given attention and positive consideration. The second premise is based on the notion that individuals often connect with like-minded others in ‘social selection networks’ based on ‘homophily’ – shared interests, values and worldviews (Aral, 2020; Prell, 2012) – known also as ‘affinity networks’ (O’Connor and Weatherall, 2019; Gehl and Lawson, 2022; see also Chapter 6 on network analysis). For the identity / profile to be effective, therefore, it needs to understand the target audience.

A second indicator of an influence strategy is the target. Target audiences range from individuals to nations and vary from individual citizens through to national leaders; they include the connecting groups, networks and influential individuals and institutions that constitute and structure societies and nations. Not all audiences will be equally significant; neither are all messages relevant for all audiences. Targeting therefore includes tailoring, which ranges from one specific message for one specific audience, to multiple differing, tailored messages to multiple identified audiences.

Both the actor and the audience can be identified and analysed through forms of monitoring, including traditional media monitoring, online and social media monitoring including social / semantic network analyses, as well as via offline methods such as focus groups, surveys and interviews. The Indicators Toolbox (Chapter 6) outlines these in more detail.

Activities

Moving from actors to actions shifts the focus from ‘who does it’ to ‘what do they do’. ‘Why’ – questions of strategy and purpose – remain pertinent.

Consideration of activities explores a range of influence-oriented communicative actions. These include mediated information campaigns, in which case the focus in on the content created and the channels used. Activities also include direct and interpersonal interactions, such as community meetings, key leader engagement and direct lobbying. Analogous terms include outreach, promotion/publicity, advocacy, and engagement. Indicators of activities are analogous to Measures of Performance (MoPs).

Analysis of activities is approached differently when planning, implementing and evaluating one’s own influence operation compared to assessing another’s influence operation.

Planning influence campaigns

When planning, the initial activities comprise research and testing to understand the target audience, develop appropriate messages, identify optimal channels for message dissemination, and undertake baseline analysis of key performance indicators for later comparison. The typical activities (See Figure 5, p.54 : Campaign Stages) include:

1. formative audience-orientated research to inform subsequent stages, including socio-cultural research, demographic and behavioural research;
2. creative design;
3. pre-testing (to check content, channel, messages), including using focus groups, test audiences, expert and peer review;
4. content production, including for paid and owned media21;
5. media relations, to organise paid and earned media placement (see footnote 7); and
6. baselining for evaluation purposes: pre-surveys; focus groups, interviews, baseline data gathering (e.g. data bases), peer and expert review.

It is noteworthy that some research methods are used multiple times through this planning stage. Focus groups, for example, may be used for initial socio-cultural research, pre-testing of creative design prior to content production, and for pre-testing for the purposes of campaign evaluation. These can include the same cohort of research subjects although typically separate cohorts are used for pre-testing the creative versus baselining, as it is considered preferable if the baseline data is derived from subjects not previously exposed to the campaign content.

Once the planning is complete and the campaign prepared, the activity consists of the launch and subsequent timing of message activity. Usually, a campaign calendar outlines the schedule listing which content appears where, and associated activities such as press releases, campaign events, community outreach and so forth. Impact assessment (see below) and ongoing monitoring occurs during the campaign, and may include mid-point or regular surveys, focus groups and interviews, with messages and activities adjusted if required in response to poor campaign impact and/or to contend with oppositional counter messaging.

All these activities, when undertaken by ones’ own organisation, can be considered Measures of Performance (MoP), which typically take the form of a list and description of outputs (e.g. press releases, social media posts) and actions (press interviews, media liaison activities, advertising purchasing schedules etc.).

Assessing influence campaign activities

Assessment of the influence campaigns of other actors is based on the identification and analysis of campaign content in the observable information environment. Typically, this is undertaken primarily through forms of content analysis and observations of activities, although the interpretation of these activities is contingent on analysis of the actors and their intentions as per above.

Content analysis methods are outlined further in the Indicators Toolbox, in brief, these methods address various questions, including:

1. What are the content types: websites, social media posts, videos, memes, speeches, sermons, meetings?
2. What are the main themes and topics?
3. What narratives does the content promote?
4. What underlying socio-cultural tropes does the content use?
5. How does the content reflect or seek to represent underlying power structures within societies?
6. How is the content disseminated: through which channels?
7. What evidence of coordination is there in the dissemination of the content: is there evidence of organisation, or loosely organised participatory networks?
8. What tactics are used to spread and boost the content, including use of bots, sock puppets, hashtag hijacking and the like?

Impacts and Effects

Analysis of impacts and effects are the most complex and most crucial aspects of influence, not least because there are varying understandings of what constitutes impact versus effect. The terms are used variously, at times interchangeably. Other terms such as outcomes, results, responses, and reception are all included in typical communication campaign plans (see, for example, Macnamara, 2018).

In this framework, we acknowledge a range of impacts and effects, from direct impacts – in the introduction, these are referred to as Measures of Impact (MoI) – to subsequent, related and emergent effects – referred to as Measures of Effect (MoE) in the introduction.

Direct impacts more closely follow influence actions, are more easily measurable, and causal links between actions and impacts are more certain. Subsequent effects include second/third order effects – additional, indirect, possibly undesirable, and/or less predictable outcomes, that can emerge over time and may extend actors distantly connected to the main campaign.

Importantly, the categorisation of direct impacts versus related effects is neither binary, nor mutually exclusive. Rather, there is a range, or spectrum, from highly direct to highly indirect.

There may be a chain of impacts, including the spread (reach) of a message, responses to these messages (e.g. engagement or counter-messaging) leading to effects: changes in intentions, opinions, attitudes and behaviours. Additionally, in complex, dynamic information environments, there may be multiple vectors that influence takes through networks of connection and association.

This idea of sequential connections from direct impacts to resulting effects underpins planning and analyses of strategic communications campaigns.

Indicators of direct impacts

Indicators of impact of influence operations involve tracking the results of the activities. For communications campaigns, this is commonly done using the forms of media and digital media monitoring outlined in the Indicators Toolbox.

Common and fundamental forms of impact analysis quantify the size of the audience/s reached and the spread of the messaging across various parts of the communications ecosystem.

Traditional media analysis focussed on the audience reach of the campaign messages. Digital media analysis also considers engagement with the messages by audiences and the ways they add to the size and spread of the content (through re-posting) and change, support or contest the content (through commenting and/or re-purposing the content). More thorough analysis will include analysis of how messages move through and across various media platforms.

Figure 5 (p. 54) lists many of the impact indicators used in typical strategic communications campaigns.

Indicators of subsequent effects

Indicators of subsequent effects shifts the focus from message content, reach and engagement (measured using content and data analysis) to that of the audiences, communities and societies. Many of the methods used for analyses of effects are listed in the Toolbox (Chapter 6).

These methods develop indicators to assess and understand types of effect, functions of effects, taking into consideration several effect factors (see Potter, 2017).

Types of influence effects include, in simple terms:

• Cognition (mental processes or the outcome of these processes);
• Beliefs (faith in something);
• Attitudes (judgements about something);
• Affect / emotions (feelings experienced);
• Physiological (bodily responses); and
• Behaviour (actions).

For effects types, influence may have several functions, including:

• Acquiring: paying attention to and remembering elements of an influence message;
• Triggering (re-activating) pre-existing beliefs, attitudes, or emotions;
• Altering pre-existing beliefs, attitudes, or emotions; and
• Reinforcing pre-existing beliefs, attitudes, or emotions.

Factors that need to be taken into account when analysing influence effects include:

• Immediacy: are effects immediate or delayed? Are the differences in immediate and delayed effects?
• Duration/durability: How long-lasting are effects? Are some effects less or more durable
• Valence: Are effects positive or negative?
• Change versus no change. Is ‘no change’ an effect?
• Scale: At what scale (micro, meso, macro) are effects?
• Directness: Which effects are direct, which are downstream (second and third order) effects?
• Intention: Which effects are intentional; which are unintended?
• Observable / latent: Which effects are observable; which are deduced from secondary observations and what assumptions/theories inform such analyses?

These factors should be taken into consideration when developing and selecting research methods and tools, identifying data sets, and when sampling populations for analysis, especially when considering effects that are indirect or delayed.

Additionally, chains of direct impact to subsequent effects may involve key individuals, social groups and networks, and large populations that build from one to another to lead to eventual outcomes. Therefore, effects analysis should consider specific yet varied effects on specific groups, and the relationships between these groups, and the overall resulting effects as pertinent to overall goals.

Tools used to develop influence indicators are contingent on these types, functions and factors of influence. Examples include:

Reputational and relationship effects between actors and audiences and other stakeholders, such as competitors are indicated through reputational surveys and interviews, content analyses, and network analyses;

Attitudes and opinions towards issues including salience (importance) of issues, and sentiments and judgments regarding issues are indicated through surveys and interviews, and through content analyses; and

Individual and group/mass behaviours are indicated through observed and recorded actions; typical indicators include voting, purchasing, attending events (protests, rallies), participating in group discussion and the like.

Proving causal effect with certainty is impossible. What is possible is the analysis of how the elements of a campaign (and contextual factors and events) contribute to an outcome. This is best undertaken by incorporating all factors and analysing them using explanatory frameworks (or theories) based in research and experience. The theories of influence outlines in Chapters 3-5, and in the companion report, Social Influence, offer such explanatory frameworks.

Effects v effectiveness

Effects measures include but are not limited to measures of effectiveness.

Effectiveness measures are the essential elements of campaign evaluation. These assess the outcomes in terms of a cost-benefit analysis, seeking to determine what aspects of a campaign should be halted, or continue, or change – and how they should change. This can be undertaken during a campaign, referred to as monitoring, and at a campaign’s conclusion, referred to as evaluation (see section on campaign planning and Figure 5, p.54).

The costs are a measure of the inputs into a campaign, included time, expertise, budget, ideas and other organisational resources. It can include less tangible resources such as social trust, reputation and relationships, which may be called on and/or exploited, perhaps damaged, as a consequence. Thus, measures of effects can include negative effects, which are counted as a cost of a campaign.

The benefits are defined by the campaign goals and the degree to which these were achieved. These include measures of desired effects, which are counted as a benefit.

The distinction between effectiveness and effects is evident in, for example, assessments of “the effectiveness of both pre-emptive (‘prebunking’) and reactive (‘debunking’) interventions to reduce the effects of misinformation” (Ecker, et.al., 2022, emphasis added) in which the effects of misinformation are calculated against the effectiveness of interventions.

Contexts

Alongside causality, context is an additional confounding complexity when assessing influence outcomes. There are so many factors that can have impacts. Some are foreseeable, and can be taken into consideration as aspect of audience analysis, such as socio-economic conditions, cultural characteristics, and political structures. But others arise out of events, which may be difficult to predict – in the case of emergencies or crises, such as a natural disaster, they may be impossible to anticipate yet have pivotal affects.

Measures of context consider these additional factors and interpret their impact on influence campaigns. Examples of questions that seek to establish the context for influence analysis include:

1. What are the relevant historical contexts, including:

a. relationships between the actors and the audiences;

b. relationships between groups that comprise the audiences; and

c. past incidents and patterns regarding issues of relevance.

1. What are the prevailing and minority views on the issues, topics, and decisions that relate to the influence operation?
2. What social structures impact on the impact of influence operations?
3. What aspects of the information environment or media ecosystem impact influence activities, such as media ownership or concentration, or access to internet/broadcast communications, or literacy levels and language expertise?
4. What other dimensions of national power, including military and economic activities, are influential? How do the dimensions of national power interact with one another?
5. What other events (unforeseen, coincidental, or because of counter influence efforts) have impacts on influence operations?

Campaign planning and evaluation – a standard, best practice approach

Frameworks from the practice of evaluation of public communication offer models for identifying and incorporating various aspects of an influence activity. Most of these frameworks are variations on a standard, widely-used best practice approach based on the integration of these stages and elements of strategic communications campaigns:

1. Goal setting;
2. Research-based audience analysis;
3. Strategic planning of overarching campaign themes and approaches;
4. Content development and selection of distribution channels;
5. Implementation and monitoring of campaign activities;
6. Evaluation of campaign – either at the conclusion of a fixed period, at a fixed date (such as an election), or at the achievement of a campaign goal. Alternatively, evaluations can be done periodically to assess and review budgets, goals, strategies and content/channel choices.

Figure 5 below follows this standard framework, and provides a list of measures and indicators that can be incorporated to assess and evaluate influence activities. Many of these indicators are outlined in Chapter 2: The Indicators Toolbox.

Organisational strategic goals

Monitoring, feedback

Planning and pre-production

Targeted audience analysis

Creative concept development

Pre-testing content

Media buying

Event planning

Pre-evaluation research / baselining data

Indicators

Surveys, focus, groups, Interviews

Peer evaluation

Implementation

Published content (publications and posts on/offline)

Paid content (advertising, boosted content) Media publicity (press releases, interviews)

Events

Community projects

Indicators (Measures of Performance) Number, timeliness of planned activities

Campaign analysis

Media mention

Circulation, reach, impressions, OTS (Opportunities to see)

Share of voice

Indicators

Media Metrics

Content Analysis

Website and social media data

Campaign Responses

Recall

Awareness

Attention

Engagement

Inidicators

Surveys, focus groups, interviews

Impacts

Awareness

Attitudes

Beliefs

Indicators

Surveys, focus groups, Interviews

Impacts

Behavior

Indicators

Observation

Stategic effects

Indicators

Observation and assessment / interpretation

Peer evaluation

Organisational

operational objectives

Preparation and project management

Organisational assets, attributes

Reputation

Relationships

Social Trust

Operational inputs

Personal

Expertise

Budget

Audiences, publics, communities, societies, competitors

Figure 5: Campaign planning and evaluation (based on Macnamara, 2018, p. 125).

Assessing malign influence campaigns – several approaches

Unlike the standardised model for communications evaluation presented on page 54 (Figure 5), there is no single accepted model or framework for the analysis of malign influence operations. There is, however, a common set of factors that many analyses draw from. Jowett and O’Donnell (2019) provide an indicative list of such factors:

• The ideology and purpose of the campaign;
• The context in which the campaign occurs;
• The identity of the propagandist;
• The structure of the propaganda organisation;
• The target audience;
• Media utilisation techniques;
• Special techniques to maximise effects;
• Audience reaction to various techniques;
• Counterpropaganda, if present; and
• Effects and evaluation.

Among the various approaches proposed, a few have recently gained prominence. The focus of these models varies, including one or more of: the actors involved; the tactics used; the content of the campaigns; and/or the impact in terms of their reach and apparent effects.

Actors

Investigations into malign influence operations will often focus on identifying the actors responsible. Attribution is a necessary precursor to accountability, to arrangements taken to limit the actors’ capability to campaign, and/or to any punitive actions (such as banning accounts, see Chapter 5).

The Online Political Influence Efforts Dataset (Martin, Shapiro and Ilhardt, 2023) suggests nine categories of actors:

• fake grass-roots organisation (known as ‘astroturfing’);
• real non-government organisations;
• governments;
• military or intelligence agencies;
• media organisation;
• companies;
• cyber espionage groups;
• wealthy individuals; and
• unknown actor.

Attribution of malicious influence campaigns can involve various indictors. For online information operations, Pamment and Smith (2022) suggests four types of indicators, referred to as ‘evidence’ (technical, behavioural, contextual, and legal/ethical) and three sources of evidence (open, proprietary, and classified sources).

The indicators suggested by Pamment and Smith (2022) are used primarily to ascertain malign influence actors’ identity for the purposes of attribution and consequently to formulate a response. These indicators can also contribute to understanding of these actors’ intentions, tactics and impacts.

A related but distinct approach is the Directing Responses against Illicit Influence Operations (D-RAIL; Miller, 2024). This identifies actors at various stages of their operations, from the initial stages in which the entity undertaking the operation develops its capabilities and creates its identity, through its intelligence, planning, implementation and evaluation stages. This is called a ‘chain of influence’.

This approach underlines that influence actors (and operations) are not simple entities but are comprised of people, infrastructures, organisations and money. It identifies entities and resources malign actors need to engage or partner with, or exploit, including: service providers and hardware suppliers; media platforms; sources of funding and financial institutions; sources of data and research companies; audiences, publics, and communities. These are all elements of and (perhaps unwittingly) actors in malign influence operations.

Campaign tactics

Analysis of campaign tactics identify the rhetorical devices and communication methods used to ‘shift’ a narrative to one more preferred (or to support a preferred narrative), and/or to either strengthen or weaken significant individuals and networks.

The Online Political Influence Efforts Dataset (Martin, Shapiro and Ilhardt, 2023) lists campaign tactical objectives:

discredit (an individual, group, or organisation);

hinder (disrupt relationships with the international community);

polarise (amplify divisive content);

spread (propagate misinformation);

support (promote politicians, policies, or movements); and

narrative framing (move political discussion in a specific direction).

The BEND framework (Carley, 2020, p.373) develops these ideas further to theorise how efforts to shape narratives and networks can be categorised. It should be noted here that the BEND framework can be applied to both malign and benign influence campaigns, proposing four positive and four negative types of tactics.

Table 2: types of sources of evidence used in naming influence actors (Pamment & Smith, 2022)

Types

Technical: uses observable traces at the level of digital signals;

Behavioural: based on actors’ known methods (also termed Tools Techniques and Procedures or TTPs);

Contextual: based on the content, the socio-political context and actors’ assumed motivations;

Legal/Ethical: considers legal restrictions on naming actors, the proportionality of doing so, and possible responses and repercussions.

Sources

Open: publicly available, including journalism, published reports, open data sets.

Proprietary: owned data with controlled access, typically confined to digital platforms, private intelligence companies, data brokers, cyber security companies.

Classified: secret information controlled by government and by extension security and intelligence agencies.

Figure 5 below follows this standard framework, and provides a list of measures and indicators that can be incorporated to assess and evaluate influence activities. Many of these indicators are outlined in Chapter 2: The Indicators Toolbox.