Chapter 5: Countering malign influence – multi-level indicators

especially as it pertains to confidence in institutions and processes of democracy and governance;

• the apparent, or at least strongly-held, confidence in mis/disinformation as a vehicle for effective political activism; and

• rising opportunities for commercial gain through forms of disinformation entrepreneurialism, in part associated with the transformation of the digital, ‘programmatic’ advertising industry.

These are largely macro-level concerns. Countering efforts (‘interventions’, a common term, used henceforth) are often undertaken by macro actors: a government, a non-governmental organisation, a news media organisation or a social media platform. However, countering malign influence also operates at micro and meso levels: the targets and tactics of interventions range greatly, from individuals and groups to societies and nations.

As these types of interventions has proliferated and diversified, emergent research has addressed three related questions: what has been tried? What has worked (and why)? How do we know?10 This section summarises this research to date. It categorises interventions according to their principal targets, namely:

• content;
• audiences / publics (those targeted by disinformation actors);
• information environments;
• platforms, and industries that support them; and
• agents (those perpetrating disinformation campaigns).

These categorisations can and do co-occur, either by design via combined interventions strategies, or coincidentally. The intentions, practices and effects of each intervention type are discussed in the next section.

Content

Content-based interventions seek to limit the visibility of problematic content, or provide a warning that content is problematic.

In the former, reducing the visibility of problematic content occurs through forms of ‘moderation’, meaning removal of accounts or content due to a violation of the rules or ‘community standards’ of the website or social media platform where the content appears. Accounts can be blocked openly, a form of blacklisting, or surreptitiously – a process known as shadow banning, stealth banning, ghost banning or (in the case of specific comments) comment ghosting. If not removed, content can be rendered much less visible on content feeds and threads by ceasing its algorithmic promotion. This process also has multiple terms: demotion, dethrottling, downranking, and deboosting.11

In the latter, labelling content with a warning can take several forms. One type is a simple graphical element indicating only that the claim in the content is false or disputed; this warning can be based on expert advice, such as an authoritative institution. Flags can be also based on responses from peers who are members of that online community.

Similarly, but beyond mere warning labels, peer networks can perform the function of social invalidation (Ziemer & Rothmund, 2024), in which problematic content is denounced in a comments section or thread, and corrections suggested. This is different from expert correction in that it is not authored or authorised by either the platform/website, does not involve formal expertise and institutional authority, and is not presented as a label – it typically appears in content threads published by members of the peer network.

Expert correction can take the form of a simple rebuttal, a narrative correction (in which the corrective information is embedded in a story), a consensus correction (which emphasises the consensus of expert opinions on the matter). Debunking is another form of authoritative correction, which includes corrective information and offers alternatives for better understanding, but also highlights the deceptive strategies employed in the problematic content.

Labelling can also include nudging strategies, in which small prompts or incentives seek to affect audiences. Accuracy nudges prompt audiences to consider content carefully; credibility nudges question the source of the content; lateral reading nudges encourage audiences to seek alternative sources to verify the content; social norm nudges promote responsible, normative behaviour, such as ‘think before you share’.

Content-based interventions are, according to surveys of research reports, the most researched form of intervention targeting problematic content (Bateman & Jackson, 2024; Puli, 2023a, 2023b; Kozyreva, et.al., 2024; Ziemer & Rothmund, 2024). Fact checking is regarded as being ‘modestly’ effective (Bateman & Jackson, 2024); content labels and corrective information are supported by some evidence as to their effectiveness, although results vary and contingent factors prevent generalisable findings (IPIE, 2023a, 2023b).

Individual-level responses vary, for example, based on a person’s attitude towards misinformation in general and their specialised knowledge of or prior exposure to the specific topic covered in the content (Kozyreva, et.al., 2024).

Responses also vary based on all of the factors identified in the chapter on micro-level influence (Chapter 2): how do people think about problems (cognitive styles); how much time have they been given; who is the source (to them); are they a member of their ingroup/outgroup and if not are they a legitimate authority; and is the influence about augmenting something they are already doing or changing altogether (which is much harder).

Audiences / publics

Interventions that target audiences and/or publics aim to increase knowledge in four related and overlapping areas: topic-based knowledge; critical media literacy; resistance to deception through forewarning (prebunking); and self-awareness and awareness of other’s perspectives.

Increasing one’s general knowledge about a specific topic increases their capacity to judge the accuracy and relevance of the information about that topic. This is the case regardless of whether one is targeted by a disinformation campaign. This basic idea seems obvious but is not always considered when assessing disinformation interventions and their effectiveness.

Critical media literacy is variously defined but can be simply characterised the ability to critically assess a piece of media content, including the content itself as well as the systems and structures that have created, curated, promoted and/or disseminated the content (McBride et al, 2023). Literacy can be further categorised to include:

• information literacy (“information, critically read”);
• news literacy (understanding the role of news in society);
• digital literacy (understanding how to use literacy tools for, for example, online image verification); and
• science literacy (understanding of how science works to, for example, develop vaccines or assess climate change) (Ziemer & Rothmund, 2024).

Inoculation, sometimes included as a form of media literacy training, is the act of pre-emptively exposing and familiarising people with the strategies and arguments used in disinformation. It is a pro-active measure base on the notion that “if people are forewarned that they might be misinformed and are exposed to weakened examples of the ways in which they might be misled, they will become more immune to misinformation” (Lewandowsky & Van Der Linden, 2021). Prebunking is a related and often interchangeable term. Typically, it presents versions of a piece of disinformation and accompanies them with refutations to the arguments and narratives therein. It can also include information about the persuasive techniques used and alert people to the ways in which information and emotions may be manipulated.

Inoculation appears more successful when:

• people have imperfect knowledge of the specific subject or of manipulation techniques,
• people are actively engaged;
• the prebunking content is based on consensus information (this links with principles of group identity from the micro section); and
• people care that they are being manipulated (McBride et.al., 2023),

Inoculation can occur prior to or after exposure to manipulative content – prophylactic versus therapeutic inoculation – with varying results. For example, research into the success of prebunking interventions to inoculate people against anti-vaccine conspiracy theories suggest that they are more successful among those who have not (yet) formed such conspiratorial beliefs (Jolley & Douglas, 2017).

Gamified inoculation uses game play to expose people to, and prepare them to confront, mis/disinformation and manipulative content. In one example, players take the role of fake news producers and learn about common disinformation techniques; experimental results indicate participants (players) effectively develop ‘resistance’ (Roozenbeek, J., Van der Linden, S. & Nygren, T., 2020).

Lateral reading, another type of critical media literacy, is a verification strategy that includes conducting additional online research to assess the trustworthiness of a claim or a source. This has been found to be quite effective in increasing the capacity to assess information. It can contribute to an increase in self-confidence and feelings of control in media use (Kozyreva, et.al., 2024).

Reports indicate that critical media literacy is among the most widely researched form of disinformation intervention (Bateman & Jackson, 2024; IPIE, 2023a; Ziemer & Rothmund, 2024). Research suggests it can be positively effective in promoting the ability to differentiate real from fake headlines and identify unreliable news sources (McBride et.al., 2023).

However, variations in pedagogical approaches mean that the effectiveness of one program cannot easily be generalised. Additionally, concerns have been raised that “critical thought can quickly become cynical thought” (Milhaildis, 2009, cited in Bateman & Jackson at 28). Media users can become cynical and distrusting of institutions, leading to them becoming believers in and advocates for conspiracy theories (boyd, 2018). Also, there are challenges in scaling up and targeting media literacy programs: “reaching large numbers of people, including those most susceptible to disinformation, is expensive and takes many years” (Bateman & Jackson, 2024, p. 24).

Information environments

There are two main types of interventions in the information environment. The first – supporting local and quality journalism – seeks to improve access to reliable information from the news media ecosystem; the second – counter-messaging – aims to provide and promote desirable information (Bateman & Jackson, 2024).

Interventions in support of local and quality journalism arise from two factors. One is the decline in local news services resulting from the drop in readership, as readers move to online sources, and advertising revenues, as digital advertising industries shift their focus to online content and targeted distribution methods (see platforms and industries, below). The second factor is the assumption that local and quality journalism fosters trust and community that fosters deliberative democracy, aids social cohesion, and challenges mis/disinformation.

There is some evidence that declines in local media lead to increases in mis/disinformation and political polarisation (Ardia et al.; Darr, Hitt & Dunaway, 2018), although empirical research assessing whether efforts to bolster local and quality media have positive effects is underdeveloped. Also, reversing the decline of local journalism is a difficult and costly proposition (Bateman& Jackson, 2024).

Counter messaging refers to the provision of truthful information. In many ways, this is similar to an ordinary political communication campaign in that it uses narratives, emotions and surrogate messengers to persuade people, but it has the added factor of being aimed at countering mis/disinformation. It differs from fact checking in that uses narrative and psychological engagement techniques to deepen the engagement with, and the appeal of, the countering messages (Bateman & Jackson, 2024).

Counter messaging is more likely to succeed when it takes into account the specific needs of an audience, tailoring content and channels accordingly – a point that is emphasised throughout this report. Success is also more likely when campaign messages, messengers, style and tone demonstrate respect and empathy (Bateman & Jackson, 2024). These success factors are dependent on a deep knowledge of the communities and their attitudes towards the issues in question, which in turn limits the scalability of any particular, specific counter messaging campaign beyond the population for which is was developed.

Platforms and industries

Several interventions target social media platforms, disinformation websites, and the industries that own, manage, support, sustain, and/or benefit commercially from them.

Some interventions are undertaken by the digital media companies that own and operate the platforms and websites. The most common of these include forms of content and account moderation mentioned above. Industry self-regulation involves digital media companies working together to set and monitor agreed standards.

Other interventions, principally attempted through a combination of monitoring, regulation and legislation, are undertaken by governments but also by an emergent field of not-for-profit agencies like the Check My Ads institute (Atkin, 2023) and the Global Disinformation Index (GDI), and commercial service providers like NewsGuard (Brill, 2024). These organisations address the ways online media and digital marketing companies profit from their capacity to profile audiences (using collected and collated demographic and behavioural data) and their ability to identify and target audiences with tailored content on online media platforms including social media and websites. This process is referred to as programmatic advertising; the technology (and sometimes the industry) is known as Adtech.

Commercial factors are highly significant factors in the size and scope of the misinformation ecosystem. Programmatic advertising is highly profitable; the size of the Adtech market is estimated to be US$600 billion and growing (Atkin, 2023). This market is largely unregulated by governments and unmoderated by the Adtech industry.

Digital advertising is dominated by large companies like Meta (the parent company of the Facebook and Instagram social media platforms), Google (owner of YouTube), and Amazon. However, the industry is complicated and opaque, involving a complex supply chain of brokers and agencies linking the demand side (those seeking to advertise) and the supply side (those selling advertising space online, or – more accurately – selling access to target audiences via online sites). Moreover, the programmatic advertising process that links buyers and sellers of digital advertising space happens automatically, unscrutinised by human decision makers, and involves billions of almost instantaneous transactions.

Under these complex and opaque conditions, it is extremely difficult for advertising companies to be aware of, let alone intervene in, the placement of their advertisements. These conditions have made it possible and highly profitable for purveyors of disinformation to provide advertising space on fake news sites or other low-quality made-for-advertising sites – sometimes referred to as ‘pink slime journalism’ (Anderson-Davis, 2024).

Some estimates claim disinformation publishers generated US$1.62 billion in advertising revenue in 2021 (for context, revenue for the entire US newspaper digital advertising market was US$3.5 billion) and that 14% of all advertising budgets were spend on made-for-advertising sites that had no editorial controls, which amounts to US$40 billion annually (Brill, 2024).

The attractiveness for nefarious actors of this poorly monitored and largely unregulated market is evident. The World Federation of Advertisers claims digital advertising is a significant source of income for organised crime (Atkin, 2023). For actors motivated by more than profits, such as those with ideological or strategic interests, the disinformation- advertising nexus offers a triple opportunity:

• revenue finances their efforts and increases their reach;
• the data gleaned from the advertising campaigns help target their audiences more effectively; and
• advertising content from legitimate companies and known brands provides a semblance of respectability and legitimacy to the disinformation it borders.

There are a number of possible interventions aimed at reducing the manipulation of the digital marketing ecosystem by mis/disinformation actors. For example, Amhad et al (2024) intervened via an information-provision experiment that sought to understand why companies advertised on misinformation websites, despite the apparent risks to their corporate reputation and the possibility of consumer backlashes. They found that most companies were simply unaware or uncertain about where their advertising was appearing. Moreover, many had not considered services (such as those offered by NewsGuard, GDI, and the Check My Ads Institute) that restrict their advertisements from appearing on misinformation sites and/or limit their advertising to appearing on pre-vetting authentic, legitimate and reputable sites.

By simply asking for details about where their advertisements appear, Atkins (2023) claims, companies can reduce the risks of their brand being associated with (and financially supporting) misinformation sites and malign actors, as well as improving the performance of their marketing campaigns.

An alternative form of intervention is based on legal and regulatory regimes that target the digital advertising industry more generally by regulating the ability of digital media companies to gather personal online data from users and increasing the requirements for platforms to moderate content. The need for stronger data privacy laws is promoted as a means to reduce the power of microtargeting and algorithmic content creation and curation.

Prominent examples of this type of intervention are the EU’s General Data Protection Regulation 2016 (GDPR) and Digital Services Act Regulation 2022. The GDPR requires commercial entities operating or targeting users in the EU to acquire user consent to collect data, and only collect the minimum amount of data needed for their stated purpose. With the GDPR, the DSA placed further limits on the use of personal data for micro-targeting of political advertising online.

Assessments of the impact of these data privacy measures is limited. There is some evidence that the use of microtargeting has declined yet the impact of this on countering disinformation is uncertain. This is in large part due to scholarship suggesting that, despite industry claims, microtargeting may not as effective as the digital advertising companies have suggested (Bateman & Jackson, 2024).

Agents

Intervention aimed at macro-level agents and networks responsible for the production and dissemination of mis/disinformation occurs in two principle forms: social media companies addressing activities on their platforms; nations acting against foreign influence operations through forms of statecraft and security policy.

Platform takedowns

Social media companies have various names for the types of activity and accounts they target for action. Meta12 (including Facebook and Instagram), TikTok and most industry observers refer to ‘coordinated inauthentic behaviour’ (CIB), Google (including YouTube) uses ‘coordinated influence operations’13 and X (formerly Twitter) referred in the past to ‘information operations’ and ‘inauthentic engagements’. Generally, these terms refer to networks of accounts, pages, groups and events that misrepresent their identity and work together to coordinate disinformation campaigns.

Most information about removal (takedowns’) of these disinformation networks comes from the companies14, who report success or at least progress in preventing and disrupting campaigns early enough to prevent them reaching a large audience. These reports are, however, unable to be verified or standardised given the restricted access to the internal research. The EU 2022 Strengthened Code of Practice on Disinformation aims to improve transparency in this area (Bateman & Jackson, 2024).

The extent to which takedowns dissuade disinformation campaigns or inspire new tactics and tools is also poorly understood. For example, a related form of coordinated inauthentic behaviour combines social media activity with inauthentic news websites, some using basic templates to mimic news sites and generative AI to produce inauthentic news content.

These sites typically produce high volumes of AI-generated content to establish an apparently authentic presence and then publish more targeted content, making them difficult to identify and address. Despite the high volumes of content evidentially produced by these networks, there is less certainty that this content is either highly engaged with or has had significant impact (Recorded Future, 2023, 2024a, 2024b).

Statecraft and security policy

Governments have several alternatives when faced with mis/disinformation campaigns conducted by or on behalf of foreign state actors, including the use of statecraft and security policy. Pamment (2022) outlines the options available:

• exposure: similar to prebunking (see above) but designed to influence a foreign actor’s decision calculus;
• attribution: assigning blame to a foreign actor, based on intelligence;
• network disruption: using cyber capabilities to disrupt internet access and digital networks;
• sanctions: imposing economic, diplomatic or other costs on foreign actors; and
• offensive operations: covert, retaliatory influence or psychological campaigns – these are restricted by many nations to operations occurring in conflict zones and supporting other direct military operations.

Additionally, nations can regulate or ban certain websites and broadcasters: the EU banned Russian state broadcaster RT; Ukraine banned Russian social media platform VKontakte; India banned TikTok. These interventions are supported by legislation and deterrence policies aimed at foreign interference campaigns; they also complement other interventions by either companies (takedowns) or through government or civil society interventions (e.g. fact checking, literacy, etc.).

Bateman and Jackson (2024) summarise the impact of statecraft and security policy on foreign influence actors.

They suggest:

• interventions are likely effective at limiting influence temporarily and in relation to specific events but only marginally effective over longer timeframes;
• sanctions are possibly effective in deterring or disrupting individual actors and commercial service providers but less evidence supports successful strategic deterrence of nations;
• deplatforming and bans have demonstrable successful first order effects, in that they reduce audience access, but may have second-order consequences such as audience migration to less visible sites and/or retaliatory bans on legitimate, factual sources of news and information; and
• exposure and attribution appears a successful tactic, especially when it receives support from both governments and oppositions in democratic nations.

Misleading indicators of mis/disinformation?

The risks posed by mis/disinformation need to be addressed through clear thinking guided by rigorous research. There are perils in under-emphasising, over-estimating and/or mischaracterising the threats of mis/disinformation. This section summarises some of main concerns arising out of inaccurate (sometimes alarmist) claims about mis/disinformation.

High levels of attention to mis/disinformation, including by using content labels, and high emphasis on the harms of mis/disinformation may lead to diminished trust in accurate information and authentic news. Experimental case study research suggests that tools aimed at improving critical media literacy, while effective in increasing the ability to identify mis/disinformation, may also have impacts on people’s trust in reliable information (Hameleers, 2023).

From a national security perspective, exaggerating the size, spread, speed, pervasiveness, irresistibility or effectiveness of disinformation campaigns has its own risks. As Belogova, et al. (2024) warn, overhyped reports of foreign disinformation can support the aims of such operations. It can nurture conspiratorial thinking: dissenters are portrayed as foreign agents or assets, open democratic debate is diminished, and domestic social ills are blamed entirely on outside forces. “Disrupting propaganda efforts by malign foreign actors is important work,” they continue, “but it must be done thoroughly, accurately, and proportionally. Exaggerating the effects of foreign influence campaigns serves only the foreign operatives.”

Misleading content?

Public commentary and journalism often greatly exaggerates the amount of mis/disinformation that audiences are exposed to; people also estimate that the majority of news they see on social media is false (Jones, 2018)15. Often focused on large numbers and seemingly impressive statistics, reports of mis/disinformation exposure need to take into account the vast amount of content online, and the many, varied sources of information that make up people’s media diets. Besides, typically mis/disinformation exposure is concentrated in smaller networks of highly motivated users representing a minority of users. In one famous example, Facebook reported that content created by the Russian Internet Research Agency reached up to 126

Chapter 5: Countering malign influence – multi-level indicators

15Similarly, Fernabach and Van Boven (2022) warn against the prevalence, and dangers, of false polarization.38

million US citizens on Facebook; less attention is given to the fact that this represented 0.004% of the content that US citizens saw in their Facebook newsfeeds (Budak, et.al., 2024; Eady, et.al., 2023).

Alongside the volume of mis/disinformation, the speed at which it can spread emerges as a major concern. This fear aligns with established axioms like that attributed variously to Mark Twain, Winston Churchill, Virgil and – here – Jonathon Swift: falsehood flies, and the truth comes limping after it; or, in a snappy version, “A lie gets halfway around the world before the truth gets its boots on”. Mis/disinformation’s velocity appeared supported by research in an influential article in the prestigious Science journal, which claimed “it took the truth about six times as long as falsehood to reach 1,500 people” (Vosoughi et al., 2018, p. 3).

However, and as the article’s authors acknowledge, the research only compared contentious content that had been fact checked — it did not compare content identified as false with content that was so obviously true it required no checking. Other research presents conflicting results: false information was re-tweeted less than science-based evidence during COVID-19 (Pulido, et al., 2020); news from hyperpartisan sites and questionable sources does not disseminate faster than mainstream and reliable sites (Bruns & Keller, 2020; Cinella, et al., 2020).

Importantly, most people’s media diets (and the internet/media content in general) are largely comprised of non-news entertainment rather than news and information. Globally, news consumption varies (Newman et.al., 2024) but is often low. Research from the US suggests news comprises 14%, and mis/disinformation 0.15%, of online media diets (Allen et.al., 2020); in France news is 3%, and mis/disinformation 0.16% (Cordonier & Best, 2021). Misinformation, therefore, “receives little attention compared to reliable news and, in turn, reliable news receives little online attention compared to everything else that people do” (Altay, et.al., 2023).

The relevance of most people having varied media diets is heightened by research into media effects suggesting that sustained, repeated exposure is required for even short-lived impact, and that competing messages from multiple sources can diminish the effect of any one message or source. Therefore, exposure to mis/disinformation is countered by exposure to other sources of news and information.

A potential counterpoint to this arises from research that used controlled experiments to assess the impact of false and misleading information on COVID-19 vaccines, suggesting that exposure to a single piece of content which discredited vaccines could impact on people’s vaccination intentions. However, this research also indicated that the most effective anti-vaccination content was not mis/disinformation, but rather was misleading claims published by mainstream sources and then distributed by well-connected super spreaders (van der Linden & Kyrychenko, 2024).

Manipulative technologies?

Social media is often the locus of anxieties about the spread and targeting of mis/disinformation. In part, this aligns with wider concerns about the role social media plays in exacerbating social problems (Budek, et al. 2024). It also reflects the academic focus on social media due to it being methodologically convenient to research (Altay, et al., 2023). The emphasis on social media comes at the cost of under-representation of legacy media and offline networks in mis/disinformation studies. It also incorrectly treats active social media users as representative of the general population, where most people rarely, if ever, post content on social or political issues on social media (Mcclain, 2021).

Another concern centres around the claim that platform algorithms are responsible for curating news feeds that promote content that is inflammatory and aimed at emotional reactions, leading to a ‘filter bubble’ effect: a concentration of mis/disinformation and a lack of alternative or balanced content. This reasoning is encouraged by the platforms’ lack of transparency about how their algorithms function. However, evidence of ‘filter bubbles’ and other algorithmic effects on people’s media diets has not been established (Bruns, 2019) and research suggests people’s news diets are typically driven by their own demands and desires, rather than (or in addition to) algorithmic curation.

Even algorithmically curated content appears to be based on past online behaviour indicating which accounts and what content people are interested in. In general, the type of content people are exposed to reflect their preferences. This includes cohorts who consume a lot of mis/disinformation, who research suggests are highly attentive to this type of content and actively seek it out (Budak, et.al., 2024).

Behavioural effects?

Finally, it is misleading to assume effects on behaviour based on exposure to mis/disinformation. It is also prudent, when assessing the impact of counter mis/disinformation interventions, not to infer behavioural consequences from, for example, critical literacy improvements, which may increase a person’s ability to spot a false news story but not change their attitudes towards vaccination (Singh, 2024). There are also concerns that critical media literacy efforts may facilitate increasing levels of cynicism and distrust

Section 1 – Influence Indicators in Principle39

about all mainstream sources of information, including that based on reliable sources and factual information (boyd, 2018).

For one, people tend to choose the mis/disinformation they consume based on existing preferences and are thus predisposed to accept it. This is especially so when that information is identity affirming, which may be readily accepted. The opposite, where information is identity threatening, is more likely to provoke resistance and disbelief (Kahan, 2017).

Also, people can hold differing, even opposing, beliefs and use these beliefs for different purposes. ‘Factual beliefs’ are used for understanding reality and operating optimally within it, while ‘symbolic beliefs’ are expressed to serve social purposes – to express and conform with group identity (Sperber, 1982) – and so therefore are not as constrained by reason or evidence. Where group identity is the issue, symbolic beliefs function as badges: the stranger and more unsubstantiated, the better (Van Leeuwen, 2023).

In this view, mis/disinformation is “not something that happens to the mass public but rather something that its members are complicit in producing” (Kahan, 2017), more “a symptom than a disease” (Singh, 2024), an expression rather than a cause of increasing polarization and declining trust. From this perspective, countering mis/disinformation includes – prioritises – addressing the conditions that lead people to become alienated and disaffected.

The role of AI in (countering) disinformation and malign influence

Artificial Intelligence applications include the use of machine learning, large language models and generative adversarial networks to create content (images and video, texts, audio including voice and music, software code), and analyse and derive insights from large data sets.

The advent and popularisation of AI raises several issues for countering malign influence. Primary concerns include:

• the ease and economy of creation of large volumes of malign fake content online (cheap fakes);
• the increasing quality and hence believability of AI-generated content (deep fakes); and
• the use of AI-supported analysis to better target audiences through identifying (a) their preferences and predispositions and (b) developing bespoke messaging, images, modes, formats, channels and timing to optimise the probability that malign influence efforts would be successful.

Cheap fakes, flooding the zone

Concerns about the increasing volume of malign fake content may be misplaced. There is already a very large amount of malicious fake content online, a product of user-generated content tools, interactive platforms and participatory digital cultures that have developed over more than two decades.

There are therefore no limits of supply of content and have not been for some time; there are limits on the demand for content based on finite amounts of audiences’ time and attention. So, more malicious fake content being created does not lead to more malicious fake content being consumed.

A caveat: the capacity for creation of large amounts of content may make it more possible for a malign actor to ‘flood the zone’, crowding out other content and dominating news feeds. This effect may be compounded if this content subsequently dominates user interactions and engagement, in turn affecting the algorithmic calculations determining content prioritisation based on this engagement.

Deep fakes, post-truth

Concerns about improving quality of deep fakes leading to more believable content and thus more people being duped are nuanced. It is evident that fake content is becoming more realistic, and realistic fake content more prevalent, because of improvements in, and increased availability of, AI-support content generation tools.

AI-detection software is also improving, which in turn prompts improvements in software that evades detection in an arms race between the creation and detection of AI-generated fakes. The contest between creation and detection is commonly used to train AI through generative adversarial networks (GANs) that pit two neural networks (one seeking to create realistic content, the other to identify this content as inauthentic) against one another so that both ‘learn’.

There are two main reasons why the rise of deep fakes may not inherently lead to more people being fooled; neither are reason for optimism. First, the quality of fake content may not be as important as the needs and motivations of target audiences. Highly motivated participants in communities based around conspiracies are likely to believe content that confirms their beliefs and disregard doubts about its source or authenticity. In this case, higher quality fakes are not more (but not less) effective.

Second, the prevalence of high-quality fake content may not lead to more fake content being accepted as fact but may lead to more fact being interpreted and dismissed as fake. In combination, these contribute to a post-truth information environment in which public truth claims are more difficult to support and sustain.

Micro-targeting, bespoke messaging

AI supports the micro-targeting of audiences based on large data sets of recorded individual preferences, habits and characteristics. The data is analysed to develop recommendations about timing, channel (i.e. which media to use) and messaging. The promise of reaching the right person, at the right time, with the right message, about the right product or service, has long been the claim of the market research. There have been and continue to be reasons to suspect that these claims may be inflated. However, it appears likely that data-driven and AI-informed research will develop targeting methods and improve results. This is the case for all types of targeted messaging, not only disinformation, but that does not diminish the concern that malign influence may become more effective through AI-informed micro-targeting.

Summary / conclusion

Efforts targeting mis/disinformation have proliferated and grown in importance in the last decade. This is largely a response to increasing concerns about threats to democratic norms and public safety, and concerns about the impact of malign interference as an aspect of information warfare.

Mis/disinformation, and therefore the attempts to counter it, occurs at various levels (micro-meso-macro) and may target different elements: audiences; content; channels and more broadly the information environment; economic systems and business models; the perpetrators and facilitators of mis/disinformation; or a combination of these.

Assessments of countering efforts use various indicators as relevant to the elements being assessed. To date, research is limited on the effectiveness of these efforts. It is difficult to draw generalisable conclusions.

The recommendations arising out of research to date are to consider all options for countering mis/disinformation, and to plan targeted countering efforts based on the conditions, contexts, contingencies, and desired outcomes.