Influence Indicators

Chapter 6: The Indicators Toolbox

This chapter summarises the indicators referred to in the earlier sections and adds a list of methods and practices used to develop indicators and analyses of influence actors, activities and effects, derived from audience analysis and strategic communications planning and evaluation approaches.

Micro level indicators

Mastery

Need for Cognition: The Need for Cognition Scale (NCS) quantitatively measures the extent to which individuals engage in and enjoy deep and effortful thinking (Cacioppo & Petty, 1982).

Uncertainty orientation: Refers to individual differences in how people manage and resolve uncertainty about themselves, their environment, the consequences of their thoughts and behaviours. Uncertainty orientation can be assessed on a scale consisting of two components (uncertainty and authoritarianism), as described by Shuper and colleagues (2004).

Analytical and intuitive thinking: There are several existing measures which can provide insight into individuals tendency to use analytical or intuitive styles of thinking. For example, the Rational Experiential Multimodal Inventory (REIm), the Actively Open-Minded Thinking (AOT) scale, and the Cognitive Reflection Test (CRT) are all commonly used indicators of individual styles of thinking.

Social reality testing: the need for mastery is recognisable when individuals seek out and share information, indicating that ideas are being tested amongst each other.

Official states of emergency: directly observing official states of emergency can provide insight as to objective threat perceptions.

Perceived threat: while threats can be objective (as above), they can also be subjective, perceived by some communities but not others. Realistic or symbolic threats can be assessed formally or informally, e.g., for example, the threat perceptions scale (Stephan & Stephan, 2017) or language use measures, such as assessing the levels of threat language in a corpus of text using LIWC.

  1. Realistic threat. Perceived threats to a group’s well-being (whether that be health and safety related, or economic and politics related) are considered realistic threats.
  2. Symbolic threat. Perceived threats to a group’s status (such as differences in values, beliefs, and attitudes) are considered symbolic threats.

Capacity to engage with content: similarly to threat, one’s ability to engage with content can be either real or perceived. Objectively, this factor can be assessed, for example, by measuring the amount of time that the content was made available to the audience. Subjectively, the audience may perceive that they did or did not have enough time to engage with the content. Example items of self-reported information engagement are reported by De Dreu (2003).

Belongingness

Social identification: refers to the commitment (emotional and psychological) that people can have to groups. Measuring social identification can provide insight into connections between people, and how subjectively important these connections are. Approaches to measuring this construct include:

  1. Self-reported measures via surveys, for example.
  2. Word counting. The written or spoken communication from group members can be an indication of connections between people. For example, the use of collective personal pronouns (“we” and “us”) can be indicative of identities. Assessing identification in this way can be achieved by natural language processing.

Group membership: people can be committed to multiple identities which have the ability to impact behaviour and influence. It is important to understand which identities are complementary or antagonistic to one another. This can be achieved using

  1. Social Identity Mapping (SIM). SIM involves the construction of a visual map which represents one’s subjective group memberships and the connections between them (Cruwys et al., 2016). This process is a means of assessing highly relevant membership-related constructs, such as the number of groups, number of important groups, group compatibility, group contact etc. SIM can be carried out in person or online using the online Social Identity Mapping (oSIM) tool (Bentley et al., 2020).
  2. Social Network Analysis (SNA) (see section on SNA, below).

Norms: refer to accepted standards within a given group.

  1. Types of norms (e.g. values, beliefs, behaviours, emotions).
  2. Must be measured in the context of the group membership (see above).
  3. Descriptive norms (what is currently done) vs injunctive norms (what should be done).
  4. Assessing patterns not only within groups, but between groups.

Autonomy

State autonomy: refers to the need to feel that one’s decisions are self-directed in relation to specific tasks and circumstances. Typically measured via self-report survey scales (for example, Van den Broeck and colleagues (2010) provide a 16-item scale of autonomous and controlled motivation).

Trait autonomy: refers to the same need, however, in a more general sense. A validated measure for this is presented by Chen and colleagues (2015), a 3-item scale which taps into satisfaction of autonomy.

Identity Leadership Inventory (ILI): the ILI is a validated inventory that assesses four components of identity leadership: identity prototypicality (representing the unique characteristics of the group, “being one of us”), identity advancement (advancing the goals of the group and preventing group failures, “doing it for us”), identity entrepreneurship (creating a sense of unity by defining the core values and norms of the group, “crafting a sense of us”, and identity impresarioship (creating a physical presence for the group which can be seen by those outside of the group, such as through events and activities, “making us matter”) (Steffens et al., 2013).

Word counting of written or spoken communication from leadership: as discussed above in relation to social identification, the use of collective pronouns in written or spoken language can be indicative of identities. Assessing the extent to which a leader/authority figure is able to cultivate a shared sense of identity can be carried out similarly through language analysis software such as LIWC, analysing the language use of leadership rather than group members.

Language style matching (LSM): LSM is a metric which measures the extent to which multiple pieces of text match in their writing style. Measurement involves calculating similarity in the use of function words (those which hold grammatical meaning, such as ‘but’, ‘with’ and ‘can’), in order to observe implicit markers of social engagement and influence (Ireland & Pennebaker, 2010). Text can be analysed for LSM using software such as LIWC, and can be carried out as follows:

  1. Pairwise comparison. When interested in directly comparing two people within a group, pairwise comparison is the appropriate method to use.
  2. One-to-many comparison. When interested in observing an individual’s fit within the larger group, one-to-many comparisons should be used.

Meso level indicators

National Solidarity: refers to the assessment of the stability of a nation-state via social cohesion. This construct can be measured using survey data, for example the World Values Survey, the International Social Survey Program, the Australian Survey of Social Attitudes (as per the Australian Consortium for Social and Political Research Inc., ACSPRI). Perceptions of national solidarity can also be assessed using qualitative methods (see methods).

  1. Attempts to violate national solidarity. Can typically be assessed via global competition over national status. For example, the International Institute for Managing Development (IMD) provides a World Competitiveness Ranking which collects survey data covering many international economies.

Civil society: the strength of civil society is an important consideration for resilience to foreign interference. Common approaches to measuring the strength of civil society include count data regarding the number of voluntary organisations and memberships of those organisations. Another approach to the assessment of civil society is Social Network Analysis (see below).

Social capital and trust: refers to broader engagement in public activities (e.g. involvement in clubs, non-political associations etc.). It can be measured via time-use surveys, such as those offered by the U.S Bureau of Labor Statistics or the Australian Bureau of Statistics. An important note, however, is that time-use surveys are generally more established in the Global North than they are in the Global South. As social capital relates to the nature of social ties, Social Network Analysis might again be useful depending on the target audience.

Social polarisation and conflict: Polarisation can be assessed using measures of voting behaviour, campaign participation, complimented by attitude measures towards specific political figures. Most significantly, decline in middle values. Measures of civil society should be balanced against indicators of social polarisation, as not all social conflict is destructive.

Trust in media: Surveys are a common form of measuring population level trust in media, and regular surveys track changes over time. The Pew Research Centre (2022) undertakes such research, largely but not exclusively based on US audiences. The Reuters Institute/University of Oxford Digital News Report (Newman et.al., 2024) undertakes worldwide research (46 media markets) using YouGov to administer online questionnaires that include questions of trust; these annual reports began in 2012. The University of Canberra (Park et.al., 2024) participates in the global research and additionally produces a report on the Australian news media market. Flew et.al. (2020) offer a more comprehensive account of the multiple analyses of trust in Australian News Media, integrating surveys by research organisations Ipsos, Roy Morgan and Endelman.

Other measures of the quality of the news media seek to assess fundamental characteristics such as the independence, diversity and professionalism of journalism. Independence is assessed, for example, using indicators of press freedom. Reporters without Borders (2024) publishes an annual press freedom index calculated using five qualitative indicators: (1) political autonomy, (2) legal protections, (3) economic constraints, (4) socio-cultural constraints (for example, attacks based on gender, ethnicity, religion), and (5) personal safety of journalists.

Diversity is (in part) assessed by considering the concentration of media ownership and the levels of pluralism in journalists and in the sources they most frequently use and/or viewpoints they prioritise. Finally, indicators of professionalism assess the normative values (like accountability and objectivity), institutional structures (e.g. editorial boards and industry bodies like a Press Council) and levels of education and professional training.

Anti-government sentiment: identifying assessing potential interactions between high-risk (i.e., radical and extremist) groups has been key to measuring anti-government sentiment. Due to these populations being notoriously difficult to sample (resulting from their inherent distrust of official institutions), measuring the expressions of these groups are typically carried out using content analysis (see below). Some other approaches to assessing anti-government sentiment include:

  1. Surveying the public. For example, the Australian National University (ANU) offers the ANUPoll, which captures changing views of governance; and
  2. Rates of disrupted violent plots reported by law enforcement. For example, the US Department of Homeland Security offer the Homeland Threat Assessment for the United States.

Social Crisis: consists of several stages according to Turner’s theory of social drama (1986). Identifying said stages and observing their outcomes in the context of a specific event can be measured using content analysis of public discourse (e.g. newspapers and magazines contain coverage of crisis events).

Macro-level indicators

Indicators of national power

National power is usually assessed at the level of one or more ‘dimensions’ of power. A common approach is to consider diplomatic, informational, military and economic (DIME) dimensions of national power, although other models add financial, intelligence and legal dimensions (DIMEFIL). A common approach to assessing dimensions of national power utilises secondary sources of data to compile, weight and combine data into an overall power measure, and oftentimes to rank nations accordingly.

The Lowy Asia Power Index serves as an exemplar. It consists of eight measures of power, derived from 30 sub-measures of power, compiled from 133 indicators. The categorisations, weightings and sources of data are summarised here (see Figure 3).16

16The complete description of all 133 indicators used is available at https://power.lowyinstitute.org/methodology/45

Figure 3, based on The Lowy Power Index (Patton, Sato, & Lemahieu, 2023).46

The Lowy Asia Power Index combines several sources of secondary data with surveys of experts.

The ISEAS State of Southeast Asia report offers another approach, surveying experts from academia, business, government, civil society, the media, and regional organisations about forms of influence: economic; political and strategic; support for global trade; and leadership regarding the rules-based international order.

Bilateral indicators of national power

Bilateral (or ‘dyadic’) indicators of national power are based on similar dimension of power to those above but use them to assess interactions between countries and reliance of one country on another.

An example of this is the Formal Bilateral Influence Capacity Index (FBIC)17, which refers to interaction volume as ‘bandwidth’ and assesses that two countries that interact more frequently and across more dimensions of activity are more likely to have opportunities to exert influence on one another. FBIC also refers to ‘dependence’ as a measure of how reliant one country is on another for its economic activity or security, suggesting countries with high levels of dependence can be more easily manipulated (Moyer, et al, 2021).

Issues-based and intangible indicators of bilateral influence

Influence can vary according to issues and how these affect bilateral relationships. Long (2022) suggests three indicators:

  • policy divergence, the degree to which goals differ;
  • issue salience, levels of importance of the issue; and
  • preference cohesion: levels of consensus on the issue within decision-making elites.
  • Policy divergence, issue salience and preference cohesion are imprecise but not impossible to evaluate, using empirical and interpretive methods and focussing on analysis of foreign policy statements and actions to infer interests and intentions, as well as assessments of domestic politics to identify key issues, key decision makers and the networks they are located within, and the dynamics of political systems.

Other intangible influence indicators are discussed in Chapter Four. These include intangible national characteristics such as national ambition and will, and credibility. Analyses of these will be informed by historical, political, cultural and ethnographic research, using various methods including those outlined in the next section.

Additional common Indicators – tools and techniques for research and analysis

Audience Analysis

Surveys and interview-based methods

Surveys: a series of questions addressing attitudes, or opinions, typically of a sample of a target population, usually collecting data from a large number of participants.

Cross-sectional surveys are observations of population at one time point, and the results may be analysed according to variables within the population, such as location or demographics.

Longitudinal surveys are observations of the same sample across multiple time points. If the same questions are asked of different samples over multiple time points, this is called a time-series analysis.

Purposive sampling surveys target a specific population based upon the criteria. An example is Delphi surveys, which address the attitudes and opinions of selected experts. The ISEAS State of Southeast Asia and Lowy Asia Power Index both use Delphi survey methods. The Lowy Index combines these with other indicators.

Surveys may include closed-ended (participants select an answer) or open-ended questions (participants provide answers, verbally or written), or a combination.

The types of participant responses may include multiple choice selections, or scale ratings. Commonly used scales include

  1. Binary or dichotomous scales, such as yes or no selections, which are simplistic but can provide useful information.
  2. Likert scales consist of a statement or question, followed commonly by a 5-point or 7-point scale range for answers, from a positive extreme to a negative extreme (i.e., “strongly agree” to “strongly disagree”). Likert scales are commonly used but impacted by biases, including central tendency bias (the tendency to cluster answers around the midpoint of the scale) and acquiescence bias (the tendency to respond positively for social desirability reasons).

17The FBIC database is interactive and available at https://korbel.du.edu/fbic. 47

Surveys are often administered online (via platforms such as SurveyMonkey, Qualtrics, or Google Forms) and over the phone using systems such as computer-aided telephone interviewing (CATI) or face to face interviewing.

Structured interviews are a means of conducting face to face or telephone surveys. Face to face interviews are carried out as intercept interviews, wherein researchers gather information directly from the consumer (i.e., a questionnaire for a customer on-site).

In-depth interviews offer rich qualitative data, consisting entirely of exploratory, open-ended questions or prompts for discussion. Participants explain their opinions and perspectives in their own words and discuss topics with interviewees.

Semi-structured interviews combine structured and in-depth interviews. A common format is to ask closed-ended questions initially, followed by related open-ended questions and prompts.

Closed-ended questions provide data that can be analysed statistically. Interviews eliciting qualitative data are explored using forms of textual and discourse analysis (see below).

Focus groups are similar to in-depth interviews but carried out in small groups, typically up to 10 people. A moderator poses questions to the group, and group members respond to the moderator as well as each other in discussion.

Sense-making methodologies and evaluation as listening, a variant on focus groups, emphasised the practice of listening to what a group of participants wants to say, rather than answering an organization’s questions. This aims to elicit and acknowledge groups’ and individual’s priorities and perspectives, and seek to contextualise understanding as located in a group’s system of sense making and modes of communication. It is a less structured and more bespoke approach, resulting in richer and deeper analyses but not easily generalizable.

Ethnography involves similar qualitative methods of research, collected through in-depth interviews as well as observations. Ethnographic research is characterised by immersion in the setting of the participants, to observe life as it happens in a field or ‘real world’ setting.

Digital ethnography, a type of ethnographic research, seeks to explore cultures and sub-cultures which exist in online spaces.

Participant observation is a type of ethnography where the researcher is immersed in, a member of, and participates in the activities of, the group. It mirrors actual, everyday experience in that members of groups need to learn about the group they belong to, but adds methods such as interviews and field diaries to formalise and systemically record observations.

Audience Response Metrics

Recall measures message memorability. Individuals are exposed to a message and later asked if they remember it and what they remember about it. Recall may be either unaided (unprompted, such as what movies have you heard about) or aided (prompted, such as have you heard about Deadpool versus Wolverine). Recall may not indicate success; individuals may not correctly recall the message’s content, relevance or purpose.

Awareness and attitudinal measures elaborate on recall, inquiring after levels of familiarity with, perceptions about, and sentiment towards, a product, service or message. A common example asks for levels of satisfaction.

Intentions asks about the likelihood of taking certain actions. Reported intentions do not equate to outcomes but provide insight into long-term effects.

Reputation refers to shared or collective perceptions about a person/brand/organization. Common examples of the collection of reputation data include companies such as RepTrak (which offers a Global RepTrak Report) and Harris Insights and Analytics (which offers the Harris Poll).

Trust (the reliance on the integrity or ability of a person or entity) is a key element of reputation, as it underpins perceptions of legitimacy. For example, the Edelman Trust Barometer offers a general measure of trust across multiple societal domains (e.g. government, media, business, NGOs, etc.).

Experimental Methods

Experimental research design involves the controlled testing of cause-and-effect relationships. An independent variable (i.e., the predictor) is manipulated by the researcher, such that different types or levels of this variable can be administered to different groups of participants.

The aim is to observe the effect of changes to the predictor on the dependent variable (i.e., the outcome). If responses to the outcome measure are dependent on the conditions of the predictor, then a causal relationship can be established between the two variables. However, for experiments to draw valid conclusions, they must be carefully designed to avoid confounding other variables which then blur the interpretation of effects.

Chapter 6: The Indicators Toolbox48

Within experimental research, there is a distinction between lab and field experiments. Lab experiments are conducted in controlled environments, whereas field experiments are conducted in natural settings, reducing the level of researcher control but reflecting, to a certain extent, ‘real world’ experiences.

Randomized Controlled Trials (RTCs) are considered the gold standard of experimental research, aimed at evaluating the effectiveness of interventions by reducing researcher and participant bias. The effectiveness is determined by drawing comparisons between the experimental condition/s and a control condition, with a subset of the target population being randomly assigned to conditions.

Randomization is particularly important in reducing selection bias. To further reduce bias, both the researchers and participants are (where possible) unaware of the condition participants have been assigned to (called blinding).

There are different types of control conditions (i.e., placebo controls, active controls, wherein the control condition engages with specified components of the intervention, or passive controls such as waitlist conditions, wherein the intervention is not administered to the control condition). Each control condition produces different effects and therefore selection of the appropriate control condition should be guided by the research goals and chosen methodology (Mohr et al., 2009).

Physiological testing is used under experimental conditions to observe responses to messaging.

  1. Pupillometric testing measures pupil dilation, which may be in response to a number of stimuli including emotion.
  2. Eye movement tracking indicates which specific pieces of content which catch their attention, how long they attend to it, or which aspects were not attended to.
  3. Galvanic skin (electrodermal) response measures an audience’s reaction to various stimuli through micro sweat gland activity, which can capture emotional arousal.
  4. Blood pressure testing can be used as an indicator of an emotional response to content.
  5. Neuroimaging measuring brain activity can be an indication of cognitive processes and attention in response to the content.

Observational trials: like experiments, observational trials involve the observation of effects of an independent variable on a dependent variable for a specified sample. However, rather than being assigned to an experimental or control condition, cohorts or panels are observed by the researcher to determine the outcomes of exposure to an independent variable.

Evaluation of influence activities campaign

Traditional media outputs18

Media mentions refers to the number of times specific content appears in the media, broadly understood to include print and broadcast news media and their online versions or equivalents. These are sourced from media monitoring agencies.

Circulation refers to the numbers of distributed copies of media products. These can be sourced from audited figures, typically from the publication itself or via an independent authority.

Reach is usually calculated as either (1) the total number of people, or (2) a percentage of the total potential audience, who saw or heard a piece of content. For example, if an audience of 100,000 (out of a population of 1 million) saw an advertisement at least once, the reach is 100,000, or 10%. There is a common distinction made between paid and earned reach (see next section) – the first being, usually, advertising, and the second via news reporting or other unpaid coverage (such as in lifestyle and entertainment programs).

Impressions / OTS refers to the number of times that a piece of content is viewed or heard, whether online, in print or via broadcast (television and radio). If 80,000 people saw an advertisement once, and 20,000 of this audience saw the advertisement six times (three times on television and three times online), the impressions are 200,000 (80,000 x 1 plus 20,000 x 6).

Share of Voice: refers to the amount of media coverage received by the content, particularly in comparison to competitors publishing content of similar focus. Calculating SOV can provide insight as to the volume of media coverage in the context of the general market, however, it cannot provide insight regarding the sentiment of said coverage (i.e., coverage may be positive or negative).

Cost per Thousand (CPM): estimating the cost-effectiveness and reach of the communication of paid content. Specifically, refers to the price of advertising per 1,000 impressions (as above).

Section 2 – Influence Indicators in Practice

18 There are inconsistencies in how some media metrics are calculated and used. We have taken our definitions from Macnamara (2018), an established authoritative source.49

Digital and Social Media

Digital Reach includes:

  1. Paid reach, which (as above) refers to the number of people who see paid content;
  2. Earned reach, which refers to publicity generated by organizations through public relations activities such as press releases and interviews. Earned Medi Value (EMV), for example, is a metric which estimates engagement with social media content via third-parties (i.e., unpaid exposure) (Carn, 2023);
  3. Owned reach, which refers to digital content created and controlled by an organization. This may include websites, official blogs, and social media accounts and content.
  4. Organic reach, which refers to content distributed by users, such as sharing and reposting (e.g. retweets (Twitter/X), reblogs (Tumblr), and regrams (Instagram). This is measured via the platform (i.e., Facebook Insights, Twitter Analytics) or via applications such as SharedCount. While people generally tend to share content they view favourably, some may share content they disapprove of, to garner outrage. Share counting cannot indicate sentiment.

Visits/sessions refers to the number of times a unique user visits a website. The session starts when the user enters the website and ends when the user stops interacting with/leaves the website.

Views/page views include measuring the number of web pages visited as well as:

  1. Time on Page or duration, an indicator of how much attention was paid to a specific page, which may be more insightful than a simple view count.
  2. Bounce Rate, which refers to the number of users who enter a website but leave shortly after entering rather than continuing to view other pages.

Clickthroughs and Clickthrough Rates (CTR) record page/website visit via users clicking on a link (which may be embedded in an image or advertisement). Clickthrough rates represent the ratio of the number of times a link is clicked to the number of times the link is presented (Lohtia et al., 2003).

Conversions refers to the percentage of people who engage in an action after being prompted by the content they are viewing (e.g. signing a petition, subscribing to a YouTube channel).

This percentage is known as the conversion rate, a generic term which can be narrowed according to the specific action being taken, such as subscribing.

Cost per Click (CPC) and Cost per View (CPV): a similar concept to Cost per Thousand (CPM), CPC refers to the fact that social media sites charge advertisers to host their content based on the number of clicks. CPC is estimated by dividing the cost of advertising by the number of clicks the content generates. CPV, similarly, refers to the average amount advertisers pay when a user watches 30 seconds (or the full duration, if this is less than 30 seconds) of an advertisement. CPV can be estimated by dividing the total cost of the video advertisement by the total number of views it receives.

Engagement (Likes, Shares, Follows) are basic metrics for the evaluation of the reception of content, the general interpretation of which is that engagement is an indicator of greater success. However, these are relatively low effort and do not necessarily indicate deep interest in the content. Moreover, incorrectly equating engagement with success is a textbook example of Goodhardt’s Law (see Introduction, p.12). A more modest approach usefully uses engagement as a proxy for attention (Spry, 2018).

Content (Posts, Comments) can be counted to provide an initial estimation of attention. More in-depth content analyses (see also next section) can assess:

  1. Sentiment, referring to the feelings and opinions expressed in content.
  2. Tone, referring to the manner of speaking, typically categorised using a broad three-point rating scale: positive, neutral, and negative.
  3. Favourability takes into account variables such as size/length of content, prominence, audience reach, and topics discussed.

Content and channel analysis

Content analyses focus on the messages created and includes consideration of lexicon, narratives, genres, imagery and multi-modal elements including textual, visual, and audio features19. Content analyses vary methodologically in terms of the size and scope of the content considered, the techniques used to undertake the analyses, and the theoretical approach.

The size and scope ranges from the very large – massive corpa, typically sourced from online or digital sources – to a single document or artefact such as the transcript or recording of a speech, or a single social media post.

Chapter 6: The Indicators Toolbox

19For simplicity, the terms ‘text’ and ‘content’ are used to include textual, visual, multimedia and multimodal artefacts – these can range from a lengthy official document to a vibrating alert to a video link on a smartphone.50

Techniques include quantitative methods, often used a sizeable corpus to undertake large scale analysis of major themes and topics. These can be undertaken using human researchers who code the content according to categories that may either be pre-determined or may emerge from the process of analysis. Also, computational methods use software with forms of pattern detection, typically based on the organisation, proximity and relationship of semantic elements in the text to identify key themes and topics, and the relationships between these elements.

Qualitative methods typically involve more attentive reading of smaller numbers of texts or artefacts with the intention of interpreting the texts in their contexts, cognisant of the cultural frameworks within which the content is created and – most importantly – consumed.

Qualitative analyses often require attention to nuance and detail, as well as specific cultural knowledge and experience, in order to grasp key references, connotations and significances. These specific analyses are often also supported by relevant theoretical frameworks, such as those discussed in Part One of this report (see also discussion of middle-range theory, p. 11).

Qualitative Approaches to content range from the largely descriptive textual analysis, focussing on identification and representation of content themes and topics, to Critical Discourse Analysis (CDA), which examines texts in their socio-political contexts.

CDA approaches seek to interpret how power (or, ‘influence’) is “enacted, reproduced, legitimated, and resisted by text and talk in the social and political context” (van Dijk, 2015, p. 467). It aims to uncover and account for the ways content utilises and/or reinforces pre-existing narratives, tropes, and assumptions about, for example, national or socio-cultural characteristics. It furthermore aims to highlight and deconstruct how these narratives, tropes and assumptions are used to shape strategic narratives about, for example, legitimacy or blame.

Narrative analysis, a type of discourse analysis, focusses on the ‘story’ that is being told, and how that story serves strategic goals. Narratives differ in this sense from other types or genres, of discourse such as speeches, regulations, policies, plans, although these may be incorporated into narratives. Narrative analysis identifies the protagonists and antagonists, their purported purposes and motivations, and their actions and outcomes.

Computational narrative analysis methods are used to identify narratives when dealing with very large texts or corpa, and especially when narratives are conveyed over multiple online sources, such as social media posts and online blogging networks (Ranade, P., et.al, 2022).

The emphasis in narrative analysis is on uncovering or clarifying how the messaging seeks to portray actors and actions in a certain light. Moreover, the objective is also to locate the narrative in a broader strategic or political context – this strategic element is what makes these narratives of interest for the analysis of influence operations; this is why these narratives are referred to as ‘strategic narratives’ or sometimes ‘influence narratives’.

Section 2 – Influence Indicators in Practice

Case study: Narrative analysis

The Digital Forensic Research Lab at The Atlantic Council (2023) analysed pro-Kremlin outlets published prior to the invasion of Ukraine. They identified primary narratives as:

Russia is seeking peace:

Russia has a moral obligation to do something about security in the region;

Ukraine is aggressive;

the West is creating tensions in the region; and

Ukraine is a puppet of the West.

Recorded Future (2022, 2024) has examined Russian information operations targeting Western governments and populations through ‘influence narratives’ which aim to divide the coalition on Ukraine.

These narratives seek to weaponise economic discomfort in Western nations by blaming support for Ukraine and policies toward Russia. They aim to turn economic pain into political discontent and direct it at incumbent governments facing elections in 2024.51

Topic Modelling is a means to organise and provide insights about text-based content. It is typically used with textual data that is larger than can be realistically analysed using manual methods. Topic modelling relies on statistical techniques based on the distribution of words that appear in the documents to uncover topics and relationships between topics.

There are various topic modelling methodologies. Unsupervised topic models, which are common, are beneficial for identifying topics without prior knowledge or guidance. Supervised topic models, which incorporate pre-identified data to influence the process, are used when certain topics are expected or of interest. Hierarchical topic models identify hierarchical relationships between topics and are used to identify topics within topics to offer a more granular analysis.

Channel analyses identify and evaluate the means (vectors) by which content moves through information environments. The focus is typically on the types and systems of communication media involved. ‘Discourse chains’ are an element of CDA that focusses on how discourses ‘travel’ from one text to others, and how they are changed through processes of re-mediation and re-interpretation. Discourse chains therefore combine CDA with elements of channel analysis, including the techno-social systems that combine communications technologies with socio-cultural systems to shape, facilitate and limit how discourses travel.

Network analysis

Network analysis examines connections between entities (social network analysis), or between ideas (semantic network analysis).

Social network analysis (SNA) is a commonly used method of analysis used to explore interactions of influence in social relationships. As such it can be applied to several topics covered in this report.

At the micro level, SNA is used to explore social influence and social selection. Social influence considers connections with others and how these connections impact opinions, beliefs and behaviours. Social selection theory similarly uses SNA to map relationships between individuals, in this case based on the notion that people select connections based on shared or desirable characteristics or behaviours, a phenomenon known as homophily.

At the meso level, SNA can be used to explore how the structures of the networked relationships contribute to social capital, and how new ideas and technologies are communicated through networks and adopted or resisted – known as diffusion of innovations theory.

At the macro level, SNA can be used to map relationships and influences in areas such as voting patterns at the United Nations General Assembly, imposition and enforcement of international sanctions regimes, and the operations of international terrorist and criminal networks.

SNA examines the relationships between actors, which may be distinct individuals, or collective units such as organisations, companies, institutions, and even nations. In SNA, these individual entities are referred to as nodes.

The relationships between nodes, referred to as edges, vary depending on the nature of the network under investigation. They may include, for example: forms of communication, collaboration or competition between organisations; trade or security agreements between nations; family ties and social relations within a community.

Figure 4 (from O’Connor and Weatherall, 2019) illustrates some simple network structures. Actually networks are not as neat. However, these indicate how different network structures can impact on how influence occurs. For example, in the Star network, the central node is by far the most influential as it has relationships with each other member of the group, and none of those have relationships with each other, and therefore have no means of influencing each other. In the clumpy network, two dyadic pairs connect the two groups (or ‘cliques’); these pairs are therefore very influential in connecting between the two groups.

Chapter 6: The Indicators Toolbox52

Section 2 – Influence Indicators in Practice

CycleWheelCompleteStar

RandomClumpy

Figure 4: Examples of types of simple network structures (O’Connor and Weatherall, 2019)

Ties may be formal (such as a funding arrangement) or informal (such as a social connection). They may be directed (such as one person following another on social media), bi-directional (such as two people following each other on social media) or non-directed (such as two people both members of the same community group). Actors may have multiple connections, known as multiplexity.

Socio-centric analyses focus on the network in its entirety, ideally mapping all the ties in a bounded community. Ego-centric analysis focuses on a main

actor (the ego) and their connections (the ego’s alters)

and the connections between the alters.

Networks may be measured as one mode (theconnections between individuals in one organisation,such as doctors in a hospital) or multi-modal(connections within an organisation and connectionsoutside that organisation, or in connection with an event).

Typically, however, most individuals and organisations

are embedded in multiple influence networks comprised

of many sources and recipients of information: many messengers and many messages. Acknowledging this moves analysis away from a dyadic perspective

(the influence A has on B) towards a more holistic ‘communication ecology’ perspective, which accounts

for multiple relationships in an open system that is

responsive to changes in its environment (see below on multidimensional networks).

Methods of data collection for SNA

SNA requires collection of two types of data: the nodes (the actors) and the edges (the relationships between actors). Primary data collection typically takes the form of surveys or interviews, done by hand or using network survey tools. Respondents identify their connections and are asked about aspects of their relationships (such as how much trust an individual respondent has in a person, or how much funding an organisation provides to another). Where surveys are not feasible, other sources of primary data like reports, organisational

charts, contracts, lists of partnerships and affiliations

are used.

Secondary data (sometimes called ‘exhaust data’),

having been collected for other purposes, may be utilised by researchers, albeit with caution regarding its veracity and completeness. Social media data outlining networks based on accounts followed, group membership, content shared and content engagement are a common examples of uses of social media data for social network analyses.

5253

For socio-centric analyses, saturation surveys aim to map entire networks. Unlike other surveys, where representative sampling may be used, network analysis requires a high response rate (ideally, above 80%).

For ego-centric analyses, name generators and position generators are options. Name generators elicit or otherwise identify an ego’s connections (alters), then using a snowballing approach identify the alters’ connections and so on. Position generators specify a role or function within a network, then seeks to identify individuals who occupy those roles, then maps those individuals’ connections.

A major concern regarding SNA data is validity/fidelity. Does the data gathered truly reflect the networked relationships? The accuracy of name generator data, for example, varies: data elicited from individuals through name-recall is less than 50 per cent accurate compared to that resulting from observations (Prell, 2012). Also, the nature of the question impacts the data. Directed ties need directed questions, such as ‘who do seek advice from’ and ‘from whom do you seek advice’. And in general, questions that offer fixed choices are less accurate than free-recall questions: asking someone to name 3-5 friends, for example, prompts responses of 3-5 ties where a more accurate answer may be more or less.

Indicators for SNA analyses

Network data is collected at the level of individual nodes and edges; it is analysed at the level of network structure, using concepts such as cohesion, partitioning and centrality.

Cohesion describes the interconnectedness of a network. Three types are:

  1. closeness, an expression of the number of links (nodes and edges) between two actors in a network – a familiar term is the ‘degrees of separation’;
  2. reachability, which outlines if the actors in a network are connected – those that are not are called isolates; and
  3. density, calculated by dividing the actual number of relational ties in a network by the possible number – the higher number, the more dense the network.

Partitioning refers to the subgroupings within networks and identifies how networks are typically unevenly connected, or ‘lumpy’. A component is a part of a network in which all actors are connected by at least one tie. A ‘clique’ is a subgroup in which all members are directedly linked to one another.

Subgroup analysis is important for the analysis of strong and weak ties in diffusion of innovations theory, which argues strong ties result in rapid sharing of (already accepted) ideas in cliques but that new information and novel ideas are more likely to enter a clique through an external, and probably weak, tie.

Centrality identifies the most prominent or influential actors in a network. The standard approach distinguishes three main forms:

  1. degree centrality, which identifies the actors with the most direct connections and indicates high levels of activity or popularity;
  2. betweenness centrality, which is a measure of an actor’s role in linking nodes that would not otherwise be connected, controlling the flow of information or influence between otherwise disconnected sub-groups and acting as brokers or gatekeepers; and
  3. closeness centrality, which measures how efficiently (via the fewest links) an actor can connect with every other node in a network and indicates how quickly information and influence can travel from that actor through the network.

SNA usually involves the use of software to collect, clean and analyse network data. Usually, the results are presented using visualisation software that present the networks as a pictorial representation of nodes and edges. Visualisation makes it easier to: overview network structures; identify key actors, sub-groups and communities; detect outliers and peripheral actors; and see changes over time.

Semantic network analysis

In semantic network analyses, the nodes (also called hubs) are words or concepts and the edges (also called spokes) are the relationships between them.

Some semantic network analyses connect words based upon relationships of meaning (i.e. semantic). Examples of this include:

generic or umbrella terms (hypernyms) and the specific terms (hyponyms) that are subtype of the generic term. ‘Red’ and ‘blue’ are hyponyms of ‘colour’; ‘navy’ and ‘sky’ are hyponyms of ‘blue’.

words that are a part of something (meronyms) and words that are the whole thing (holonyms). Fingers (meronyms) are part of a hand (holonyms).

Other semantic networks analyses are based on co-occurrence of terms. This includes identifying keywords of the most popular terms in a text, calculating the frequency with which these terms co-occur with other key terms, and analysing these frequencies and proximities to identify and cluster themes in the network. This can often be done using computational analyses to identify networks of themes in a larger corpus.

Multidimensional networks

O’Connor and Shumante (2018) emphasise the multidimensionality of networks in practice: organisations and individuals are embedded in multiple networks and multiple types of networks. They suggest two main types of networks: (1) those that indicate connections between actors based on activities: who does what and with whom do they coordinate their activities; and (2) those that identify how messages, information and ideas move through communicative connections, and how these messages develop meanings within networks.

The first type is based on social connections between actors and is typically defined as social network analysis, as per above. The second is based on the semantic (meaning-based) relationships between words, images (and other meaningful symbols) and is therefore referred to as semantic network analysis, or – when the emphasis is on the transmission of ideas or messages – flow networks.

Flow networks “decay rapidly” (O’Connor & Shumante, 2018). If new messages are not transmitted, the flow ceases. If flow networks are persistent, they can result in more robust forms of connection. Affinity networks, based around homophily (shared understandings, motivations and attitudes) can evolve from flow networks, and incorporate social connections based on agreements about ideas.

Semantic networks are more widespread and maybe more resilient than affinity networks. (An apt analogy might be fans of a star sports celebrity versus a sporting team’s brand, image and reputation.) There are clearly relationships between affinity groups and semantic networks, which underscores the point about the multidimensionality of networks analysis.

The key points here are (1) network analyses can include consideration of multiple networks, multiple types of networks, and the relationships between them, and (2) multidimensional network analysis is a useful approach to in-depth audience analysis, measures of impact and effect, and assessment of malign influence operations.

Case study: Russian Star / Affinity networks in practice

Russian influence operations active in the United States in the context of the 2016 Presidential election campaign appear to have sought to use star/affinity networks. According to O’Connor and Weatherall (2019), they created community pages on Facebook based around shared affinity (such as gun rights, or abortion rights). This positioned them as having an alignment with their group members priorities, attitudes, beliefs and values. Such an alignment increases levels of trust.

As administrators of these community pages, they prioritised their own posts on their pages’ content feed. Posts submitted by other community members were made more difficult to find, obscured behind a series of inconvenient links. They could communicate to the whole group more easily than group members could communicate with each other, creating a star network in which they were the central node.

This combination of trust and network centrality could then be used to more effectively influence the network.