Chapter 1: The Principles of Influence Indicators

What is influence?

This report is the companion to Social Influence – Integrated Influence for Engagement and Resilience (West et al. 2023), which outlines the foundational concepts of influence. The Social Influence report emphasises key principles that contribute to an enhanced conceptual understanding of what influence is and how it operates:

1. Influence operates at multiple levels or scales: micro, meso and macro;
2. Influence is a product of co-created meaning making, including the interpretation by audiences and publics of the messages and actions of others; and
3. Influence is understood as attributes, actions, and/or effects.

Multi-level influence

Influence operates at various levels: from the individual (micro), through social groups and networks (meso), to the institutional, national and international (macro). The operation of influence at these levels is developed in Chapters 2-4.

These levels are not clearly delineated categories: the lines blur between micro, meso, and macro. Nor are these levels mutually exclusive. In practice, influence operates at various scales of magnitude regarding the numbers of people, the complexity of the socio-political environment and even the time involved.

Complex, multi-level systems require multi-scale integrated analyses. The range of indicators and measures should be considered as a suite of approaches, methods and tools that in general are part of a coherent whole and which may be applied selectively according to the bespoke requirements of a specific project, based on the research/applied question being addressed and the operational requirements this research is supporting.

Just as in Figure 1 (see p.8), individual trees may provide sources of interpretation and information, the full forest will be more easily gleaned if multiple sources are consulted and triangulated.

In fact, there are many indicators that can be applied at various levels or scales of analysis, and many that require consideration across multiple levels and scales. This is because influence typically operates in complex systems that involve individuals, groups, societies, and institutions.

Influence is relative and relational – it requires consideration of many influence actors and their relative influence capabilities and activities, and it requires consideration of the relationships between those involved in the production of influence.

Influence occurs – appears via, flows through, and impacts on – social networks. These networks operate online and offline. They include technological and communication networks, as well as groups and communities based on shared socio-cultural connections and identities (e.g., languages, religious affiliations, etc.).

Co-creation of meaning

Second, influence occurs as a result of co-creation of meaning by those seeking to influence through forms of persuasion, manipulation, or coercion1 and those who interpret and act upon these influence efforts.

The co-creation of influence underscores for influence planners and evaluators the importance of understanding the target audiences, as they are co-contributors of the outcomes of influence actions. Understanding target audiences occurs at micro, meso, and macro scales and includes consideration of how:

• cognition and emotion affect decisions by individuals and groups;
• individual and group decisions are embedded in social relations;
• social groups are formed, develop, and are influenced through processes of cohesion and conflict;
• social power (including key stakeholders and elites) impacts on group and individual decision making; and
• groups and populations impact (to greater or lesser degrees) political decisions, including those related to national security and international relations.

Multiple measures of influence

Finally, the term influence can and has been used in many ways. It can conceptually refer to attributes, actions, and/or outcomes. The range and use of indicators (or ‘measures’) of these – and the contexts in which they occur – are the main subject of this report.

Influence as an attribute refers to the influence (and the resilience to influence) that an actor or group may possess by virtue of that actor’s characteristics, capabilities, relationships, and positions of relative power. Influence is an attribute when a person, group, institution, or nation is regarded as being ‘influential’. It is the potential to influence, converted to actual influence through action.

Influence attribute indicators identify those characteristics, capabilities, relationships, and positions of relative power that an actor or group possesses, as relevant to the situation. In this case, influence indicators express the potential for an actor to exert influence, to be influential, and is analogous to a summation of that actor’s standing or esteem within a community, its centrality within a network, or its power over others. This aspect of influence asks: how influential is an individual, a group, a nation, and how is this indicated?

Influence as actions refers to something that an entity does. Typically, the types of influence activities are designed with the intent to influence how another actor thinks, feels, or acts. The focus in the first instance is on the actions themselves, and foregrounds the activities undertaken, including the strategies, tactics, tools, and techniques used in influence efforts (or efforts to resist or mitigate influence).

Influence action indicators identify, categorise, and analyse actions undertaken by influence actors. These indicators describe what is done, why it is done (where possible), and reactions and responses to these actions. This includes descriptions of the type and content of influence messages, the means used to communicate these messages, and follows the re-distribution, re-interpretation, and re-purposing of these messages by individuals, groups, networks, and societies.

In formal program evaluation, these are often referred to as Measures of Performance (MoPs) or simply as ‘outputs’. Typically, these are acts of communication and interaction, mediated (i.e. online and social media, mass media such as radio and television) and/or interpersonal and direct interactions. However, all actions – including exercises, operations, procurements, publication of strategies and reports – have communicative effects. These effects are dependent on the interpretation of those observing the action.

MoPs focus on those activities that are planned and carried out as elements of a campaign or operation. In a typical communications campaign, for example, such activities include, speeches, publications, media releases, social media posts, public events, key stakeholder engagements, briefings to the news media, and much more.

Actions also include activities that are not considered to be deliberate aspects of an attempt to influence but may be consequential, if unintended, in the sense that these actions are interpreted as meaningful by others. These are not always included in the Measures of Performance of an influence operation, which can lead to incomplete or suboptimal analyses, as actions that are not formally included in an influence operation may, and often will, have the largest impact. The expression attributed to Ralph Waldo Emerson sums it up: “your actions speak so loudly that I cannot hear what you are saying”.

Influence as outcomes refers to the effects that result from influence actions. These outcomes vary greatly in terms of their scale: they may include an individual choice, a group response, a national decision, or an international agreement. They vary in terms of how consequential they may be, from a choice to donate a few dollars, to a change in the levels of inter-group social conflict, to a national decision to go to war. Outcomes also include second and third order effects, either foreseen or unanticipated, that increase the outcomes to those beyond those directly targeted.

Influence outcome indicators are often considered the most important. For planning, monitoring and evaluation of influence activities, the outcomes are what really matters. Influence outcome indicators are also typically recognised as the most difficult to ascertain with certainty. Problems arise from: the access to quality, relevant, and sufficient data; the complexity of the situations in which influence operations occur (not least being the role of multiple actors); and the questionable causal relationship between action and outcome. This report seeks to address some of these problems by providing means to undertake analysis of influence outcomes that, while not aiming for certainty, can aspire to increased confidence.

In formal evaluation, the terms outcome, impact and effects are often used, often interchangeably. Here, we distinguish between Measures of Impact (MoIs) that are more proximate and directly attributable to the actions undertaken, and Measures of Effect (MoE), which are typically related to either the goals, or an identified ultimate end-state, of an influence operation.

MoIs seek to identify, describe and/quantify the results of the actions undertaken, and typically follow MoPs. For example, if the MoPs include the details of the press releases issued, social media posts published, and speeches given, then the MoIs seek to identify the details of how the press release was covered including the audiences of that coverage, the reach and engagement of the social media content, the immediate audience and subsequent reporting of the speeches.

MoEs can be characterised as either cognitive (on knowledge, opinions and/or attitudes) or behavioural (on behaviour). They can also be characterised as immediate and direct (first order) or indirect, unintended and/or occurring in a (much) longer timeframe.

The distinction between MoIs and MoEs is at times unclear. MoIs tend to be more measurable and directly related to the activities undertaken. MoEs tend to be (and should be) more closely focussed on the desired outcomes2; they also tend to be much more difficult to measure. It is even more difficult to demonstrate casual relationships between the actions undertaken and the cognitive or behavioural effects observed, as there are too many variables, and data sources are typically inadequate.

Yet, MoEs are essential aspect of planning and especially evaluating influence activities. A main aim of this report is to outline how to best address the confounding yet crucial issue of how to better predict (for planning) and assess (for evaluation) the effects of influence activities, using MoEs that are focussed on quality and variety of data and sound analysis based on robust theories of influence. In chapter 7, planning and evaluation of influence activities, including the use of MoIs and MoEs, is discussed in more detail.

Measures of Contexts (MoCs) take into account environmental, accompanying and contingent factors that have had a perceptible and noteworthy or significant bearing on events and outcomes. Examples of MoCs include unforeseen substantial events, like natural disasters or major political incidents. MoCs also include ongoing characteristics of the information ecosystem that impede, advance and/or shape influence messages, such as the actors and structure of the news media system). At the broadest level, MoCs can consider the socio-cultural and political-economic systems and in which actions are taken. Such factors can shape how actions are interpreted and responded to.

MoCs may also include analyses of counter-messaging by opponents and other narratives promoted by those simply competing for attention. If a counter-message is a direct response to one’s influence operation, this might also be regarded as a MoI; if the elicitation of counter-messaging is a specific campaign goal, then the analysis of such messages would be a MoE. As emphasised earlier, analysis should be framed by the aims of a campaign or the main concerns about influence activities under observation. ‘Why’ matters.

What are ‘influence indicators’?

The term ‘indicators’ as used in this report refers to (1) various sources and forms of information and representation that, (2) subject to careful analysis and interpretation, (3) enhances understanding, knowledge and insight and (4) supports improved decision making.

The various sources and forms of information and representation3 are sometimes referred to as the actual, ‘raw’, indicators. There are many; they vary greatly.

Sources include pre-existing databases, archives and records, key informants, specialist experts, local communities, online users, and many others. The forms indicators take includes quantifiable measures such as the size and the characteristics of an audience, or a community group, positive or negative opinion as represented in opinion polling, the results of large-scale survey or interview-based studies, to name a few. Qualitative indicators can include the key terms, themes and main narratives of an influence information operation, the core identities and/or values that define a social group, the existing political cultures and institutions that may shape individual and group decision making, and many more.

Figure 1 provides an overview of the process and practice of influence with an idealized cross-section of the earth as a metaphor. Three layers of earth lay below ground and are therefore not directly observable: these reflect the micro, meso and macro layers of influence, respectively. Above ground, trees represent the products of those processes, where deep roots below ground give rise to that which is observable above ground. Each tree represents a different method via which the primary concepts can be indicated and assessed. In order to see the forest, the methods of multiple trees must be employed.

These themes are expanded upon throughout the report and the Influence Indicators Tool Box outlined in Chapter 6 includes a comprehensive list of indicators.

Careful analysis and interpretation refers to the methods that are applied to the information in order to derive conclusions from that data that may be used to develop understandings or explanations that have a wider application. Analysis can include quantitative approaches such statistical analysis, network analysis, and calculations of patterns over time, or using other categories such as demographics, locale, socio-economic status and the like. Qualitative analysis includes methods such as discourse analysis, thematic analysis, socio-cultural and political analyses of power relations, and historical analysis of, for example, strategic and political cultures.

Analysis is never objective, and conclusions are never certain. All analysis includes choices and limits. Rather than objective, analysis is (and should be) purposeful. Rather than aspiring for certainty, conclusions can offer degrees of confidence and/or likelihood.

The point of the analysis is to ‘contribute to enhanced understanding, knowledge and insight, supporting improved decision making’. This refers to the role influence indicators play in the planning, implementation, monitoring, and evaluation of influence activities, principally by enhancing target audience analysis and situational awareness. This can take the form of an assessment of the needs, attitudes, and values of a community at whom an influence campaign is directed, for example, or an appraisal of the key influential individual decision-makers as pertains to a particular influence outcome. In the case of assessments of impacts and effects, emphasis is on apparent changes (in behaviour, emotion and/or cognition) that are observable, discernible or inferred, and which may relate to influence activities.

Quality Indicators and Analysis

In the subsequent chapters, types of analytical approaches are outlined in more detail. Regardless of the methodological approach taken, there are some fundamental considerations that apply to all research and analytical methodologies.

There are no perfect indicators, no ideal methods, and no flawless analyses. However, the following questions act as a set of guidelines and guardrails, steering analysis in directions that are productive and useful, and avoiding practices that can contribute to inadequate or misleading results.

Purposefulness – The first and most fundamental question is: do the indicators address the target, goal, or problem set? (This is addressed below in terms of Goodhardt’s law.)

Representativeness – Are the sampling methods sufficient to represent the population, time period, range of activities, or other elements to the extent required?

Reliability – Is the data a consistent representation of reality (fidelity)? Are the methods of measurement sufficiently precise and accurate? Are the sources of data trustworthy?

No data set will be exhaustive, but limited data leads to limited conclusions. Much analysis utilises convenience samples – data that is already available or is easily accessible. Common errors include focussing solely on content that is online, or populations that respond to online surveys, or utilising data from reports generated for other purposes. In studies of social and political issues in foreign contexts, a common limitation is not examining texts such as newspapers produced in the local language but rather relying on material produced in the major world languages or crude translations. Inferring generalisable conclusions from such restricted samples can be misleading.

This is not to argue that studies should not be undertaken, or do not contribute significant insight, even if they have limitations. There are inevitable and oftentimes unavoidable limits on the data that is available. Security protections, commercial confidentiality, personal privacy and other legal and ethical considerations impose limits. Also, there are practical and logistical realities, including constraints on time and budgets, as well as levels and types of expertise and experience, that limit the amount of data that may be gathered (or generated), collated, cleaned, and analysed.

The causality conundrum

Influence analysis typically aims to identify and examine plausible and demonstrated causal relationships between influence actions and effects. One of the most accepted approaches to this is what Robert Merton (1949) influentially termed middle-range theory.

Middle-range theory combines theory with empirical research, drawing on observable data and utilising analytical approaches that allow for a level of moderate abstraction, between the specific and the universal. This analytic approach highlights problems with causal arguments based in grand analysis of social systems that inadequately consider particular and contingent social behaviour, organization, and change. While all forms of analysis involve a level of abstraction, studies following a middle-range approach emphasise the need for conclusions to be modest and grounded in observed data of particular cases (Hedström, P. & Udehn, 2009).

An example of employing a relevant middle-range theory approach in the analysis of influence is an appreciation that, in general, all audiences are active, co-constructors of meaning in all types of communication. This type of approach is both “explanatory and exploratory rather than designed to generate precise universal relationships [building] a case for a causal relationship at some broader level even if the precise effect a variable will have in any individual case is unknown” (Mazarr, 2022, 24).

Particular outcomes can be more precisely explained by combining the supportive framework of general middle-range theories with more detailed, specific accounting of the circumstances leading to, or at least associated with, said outcomes.

Hazards and limitations of indicators

Useful, quality indicators will provide means to analyse and understand a situation, action, actor, or outcome. They should:

• be informative, in that they attempt to show things as they are;
• be as directly relevant as can be to the target, goal, purpose, or problem set under analysis;
• not be designed to make a point, demonstrate success, gain approval for a previously determined plan, or gain reward; and
• not be used only because they are readily accessible.

Among the most common errors that plague analysis and especially evaluative analyses of performance and outcomes are those related to the selection of the data used to inform the analysis and the associated assumptions made about this data.

Availability bias / Measurement inversion

A frequent and understandable error can arise out of availability bias, wherein the data that is conveniently available or easily accessible is relied upon. This is also referred to as measurement inversion – when “available metrics are used whether they are the right ones, rather than the factors that are the most important” (Macnamara, 2018, 145-146; see also Hubbard, 2007).

This is understandable as (1) frequently data is difficult or expensive to source and (2) it is not always or even often clear to discern which data or factors are more, or less, important. Nevertheless, it is important to be alert to the dangers in the idea that ‘what can be counted, counts’ and seek to identify those factors and data that offer better explanatory potential.

Confirmation bias

Confirmation bias occurs when evidence is sought only to support an established position or a desired outcome, or to buttress reasons why someone’s intuitions are correct (Mercier, 2022).

Sometimes, this is deliberate. In prosecutorial arguments such as those made in adversarial debates or legal trials, for example, opposing arguments will select evidence and structure arguments that support their case. (This is contrasted with the inquisitorial approach, which seeks to establish the truth of a matter.)

Confirmation bias may also be inadvertent. It may be a product of typical desires to be correct, to ‘believe too much in a favoured hypothesis’ (Klayman, 1995) and to achieve one’s goals by convincing others that we are correct. It may also be embedded into the approaches used in analysis, including the sampling method, the empirical methods used to generate data, and the premises of the analysis. We tend to see what we look for, and we are all positioned with pre-existing perspectives on and relationships to the subjects of our inquiry. No one is objective, although we can at least be aware of our subjectivities and factor that into the analysis.

Goodhardt’s Law

Goodhardt’s law, associated with economist Charles Goodhardt, further addresses the risks associated with reliance on convenient, quantitative measures. Goodhart’s Law states that “when a measure becomes a target, it ceases to be a good measure.” Psychologist Donald T. Campbell, around the same time, echoed this sentiment, claiming “the more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the apt it will be to distort and corrupt the social processes it is intended to monitor” (both cited in Harford, 2021).

In other words, when we use a measure to reward performance, we provide an incentive to manipulate that measure in order to receive that reward. This can sometimes result in actions that reduce the effectiveness of the measured system while paradoxically improving the measurement of system performance (Stumborg et.al. /Centre for Naval Analyses, 2022) It is more clearly explained with examples of Goodhardt’s law in practice:

In an oft-cited historical example, French colonial authorities in Hanoi sought to reduce the rat population. They offered a bounty on rat tails. The measure for the authorities (tails) became, for the bounty hunters, the target. Enterprising bounty hunters bred rats, amputated their tails and kept them alive to breed further. Rats bred in the countryside were shipped into Hanoi for the bounty. The bounty hunters received their bounty; the rat population increased.

In contemporary examples involving websites and social media platforms, metrics indicating reach (views, page views), and engagement (likes, shares, comments) are used to indicate the success of an online communications campaign. This can result in (1) the purchase of engagement from ‘click farms’ or similarly inauthentic online actors via the ‘dark PR industry’, and (2) the pursuit of these ‘vanity metrics’ through the creation of content that attracts attention and engagement, regardless of whether the campaign’s target audience has been reached.

These examples illustrate how measures can reward actions despite those actions not leading to the achievement of the campaign goal or target. Worse, even when those actions result in the opposite of what is desired, they be measured as success – and rewarded as such, providing motivation for actors to either deliberately game the system or unwittingly act against a campaign’s goals.

As contemporary economist Tim Harford (2021, 59) explains:

Goodhardt and Campbell were onto the same basic problem: a statistical metric may be a pretty decent proxy for something that really matters, but it is almost always a proxy rather than the real thing. Once you start using that proxy as a target to be improved, or a metric to control others at a distance, it will be distorted, faked, or undermined. The value of the measure will evaporate.

Large data bias

Lastly, it is prudent to raise some warnings about the impact of large numbers. Especially in analysis of large populations and large online and social media metrics, large numbers are commonly cited as evidence of impact and effect of influence campaigns.

One example is the concerns arising out of reports that Russian-backed Facebooks posts reached 126 million Americans in the lead up to the 2016 US Presidential election. A first point to make is that, on the internet, there are a lot of big numbers: there are a lot of users, and a lot of content, online. So, 126 million may or may not be, relatively, large.

A second point is that the large number may occlude smaller, more relevant results. In this case, according to Eady et.al (2023, 1):

exposure to Russian disinformation accounts was heavily concentrated: only 1% of users accounted for 70% of exposures. Second, exposure was concentrated among users who strongly identified as Republicans. Third, exposure to the Russian influence campaign was eclipsed by content from domestic news media and politicians. Finally, we find no evidence of a meaningful relationship between exposure to the Russian foreign influence campaign and changes in attitudes, polarization, or voting behaviour.

Big gets the attention, but influence is typically small, marginal. Small does not mean insignificant; slight changes can be decisive. Small numbers are also worthy of more attention because, while significant, they can by virtue of being small be difficult to see, or for their importance to be recognised. Thus, small numbers may be overlooked even when they are crucial or even decisive.

Conclusion – the use of indicators

Indicators are indicative, not definitive.

Without indicators, decisions about planned activities, resources allocations, training needs and the like are at the risk of being made based on untested assumptions, the habits of past practice, the convenience of easily available options, or the demands of an unaccountable authority.

With relevant, accurate indicators, assumptions can be tested, options considered, new scenarios explored, changed circumstances taken into account, and, ultimately, crucial – and often costly – decisions guided by suitable information and careful analysis.

This principle – the indicative and illustrative, not deterministic, nature of influence indicators – applies across the various levels of influence indicators that are examined in further detail in the following chapters. It follows from this that using a number of different indicators should result in more useful analysis.

Another principle of influence is that the different levels at which it operates – micro, meso and macro — are interrelated and mutual constitutive. Micro-scale influence affects decisions that are embedded in social groups and may have macro-scale effects. Individuals are embedded in groups, communities, and societies; nations are made up of institutions, constituencies, populations, economic systems and infrastructure networks. The relationships between these various levels of influence shape how influence operates. Therefore, while the following chapters are separated into three different levels, they should be considered as different aspects of integrated systems of influence.