Big data and national security: A guide for Australian policymakers

The reach, impact, volume, and speed of data has changed the way states, businesses, groups, and individuals communicate and make sense of the world.[1] Data is remodelling society’s relationship with government, changing participation in the economy and access to services, as well as challenging trust in social, commercial, and government institutions.[2] Big data is also redefining national security[3] and the way nations protect individual rights and freedoms.[4]

Big data is loosely defined as data that is too large to be manually processed. It allows literally millions of pieces of information — from location points, financial transactions, and social media profiles, to medical files and video streams — to be brought together and analysed. The analytics and technologies used to derive value and insight from this data,[5] such as artificial intelligence (AI) and machine learning (ML), should be considered part of big data.This paper is in two sections. The first outlines the foundations of the big data landscape. It examines how the ubiquity of technology — our daily reliance on data infrastructure and our participation in the landscape — forms the backbone of the new economic, political, and social power of institutions. The second section outlines how big data technologies are adding new threats to national security[6] and exacerbating existing ones. This comes at a time when Australia’s strategic circumstances are more malign than in recent decades.[7] In this environment, Australia needs to leverage big data and emerging technologies for strategic advantage.

Big data and emerging technologies have transformed the global information landscape and national security operating environment. Big data transfers power to organisations that hold the most data, control global data and information flows, and provide digital connectivity. It creates oligarchies among companies whose technology is the most ubiquitous.

These powerful private entities have created a new information and infrastructure landscape with minimal oversight from national governments. In this environment, technical developments occur much faster than regulation. The pace of development, combined with the complexity and interdependence of technologies, along with issues of global reach complicate the work of legislators and regulators, who are largely not digital natives.

Big data arose from technical advances in storage capacity, processing speed, and the declining cost of data collection and analysis as well as the move towards understanding data as continuously collected, almost infinitely networkable, and highly flexible.[8]

Big data “is less about data that is big than it is about a capacity to search, aggregate, and cross-reference large data sets”[9] to analyse and derive insight,[10] usually to create economic value.[11] Previously, databases were unable to simultaneously deal with what are known as the 3Vs of big data: volume, velocity, and variety.[12] But increased computational power, new database designs, and distributed storage now enable collection and analysis of big data.[13] The definition has subsequently been expanded to 5V to include veracity (determining uncertainty and inconsistency in data) and value (gaining insights into and from data).[14]

There are three features of big data with unique and significant implications for national security: data abundance, digital connectivity, and ubiquitous technology.[15] (See Figure 1). These features create a big data landscape that concentrates the data, technical capabilities, and analytical capacity that are increasingly essential for functioning societies.

1. Data abundance

“Data abundance” refers to the vast and rapidly growing volume of digital information that exists in society.[16] By 2020, the number of bytes (units of memory size) in the digital universe was 40 times the number of stars in the observable universe.[17] There are three primary locations where digital content is created: the core (traditional and cloud data centres), the edge (enterprise-hardened infrastructure, such as cell towers and branch offices), and the endpoints (personal computers, smart phones, and Internet of Things (IoT) devices). The summation of all the data collected in these locations, whether it is created, captured, or replicated, is sometimes called the global datasphere.[18]

The amount of data is not the whole story. The ability to represent the interactions of daily social life in online, quantified data — datafication — has increased dramatically.[19] While datafication is not entirely new, it is increasingly sophisticated and nuanced. Big data has made data collection about human interactions — including aspects that were previously unrecorded — omnipresent. As people go about their daily lives, their activity leaves digital footprints,[20] with data constantly created by their movements and activities,[21] making it virtually impossible to exist without leaving a digital trace.[22]

Much of this increase in datafication has been driven by the development and adoption of smart phone technology. On average, global users interact with their mobile phones hundreds of times per day. The average Australian spends 5.5 hours per day on their phone.[23] Much of this data is generated through the daily habits of social media, shopping, searching online, and fitness tracking, but large volumes of data are also created and collected by machines to track the daily lives of users.[24] (See Case Study 1).

This abundance of data (and sometimes the absence of it) enables those who collect it to make inferences about the beliefs, values, preferences, psychological state, and intimate details of those who produce it, including people’s feelings and vulnerabilities.[27] These inferences are made about individuals, often without their knowledge, by the aggregation of data collected from seemingly mundane activities. In short, big data has exploded the scope of personal and personally identifiable information.[28] Some data is individualised and some of it is collected in so-called “anonymised” data sets, although almost all of it can be re-identified to the individual level.[29] It is possible to build an increasingly comprehensive picture about people and things from data alone, even if that data is anonymised, as illustrated in Figure 2.  

Datafication is an inherently commercial activity. Most data is created by and resides in the private sector, in tech companies, and with data brokers, as does the analytical capability to make sense of it. Data broking companies aggregate personally identifiable information about consumers from different sources, then match, license, and sell that information.[30] Commercial data sets are created, bought, and sold by third party data brokers, acquired by purchase from private companies and by trawling public information sources.[31]This constitutes the big data economy and the commercial value of data is linked to whether it can be attributed to an individual’s identity — the more data obtained, the more granular a profile it produces.[32] The deeper the level of detail, the more targeted individual advertising can be. Large commercial data sets can be purchased by anyone, including in some countries by state security services and government organisations.

To provide a sense of the size of this market, of the 4000 data brokers globally, one of the largest, Acxiom, is said to have 3000 data points per person for 500 million consumers worldwide.[33] The global data broker industry was estimated to be worth US$178 billion in revenue in 2018, and PricewaterhouseCoopers market researchers suggest that by 2025, the global data economy will be worth more than US$400 billion.[34] Estimates suggest that Google, Amazon, Microsoft, and Facebook alone store at least 1200 petabytes, or 1.2 million terabytes, of data between them.[35] However, the true size of the market is unknown, as brokers operate with little or no transparency in primarily unregulated spaces.[36] Large data stores held by commercial actors are vulnerable to hacking and exploitation by nation states and criminal actors,[37] as seen by the breaches at Equifax, a multinational consumer credit reporting agency, in 2017; the Australian National University in 2018; and data collection company Oxydata in 2019.[38]

2. Digital connectivity

Digital connectivity is the ability to connect people, places, and ideas through virtual networks.[39] Digital connectivity includes billions of sensors and devices around the world connected to the internet.[40] It includes the relationship between things and people made possible by networked technologies and various platforms,[41] such as computers, mobile phones, and the Internet of Things (IoT), enabling previously unconnected agents to connect.[42]

The number of devices connected to the internet has increased exponentially over the past 20 years. Estimates vary between 100 and 200 billion devices connected at the end of 2020.[43] When individuals use devices to send, receive, broadcast, and share information, digital connectivity is most visible.[44] Less visible is the vast network of billions of sensors in sectors such as business, manufacturing, healthcare, retail, security, public places, transportation, and in “smart” home devices.[45] In 2020, machine-to-machine communications accounted for 40 per cent of the total traffic between sensors and this ratio will continue to rise. Unprecedented digital connectivity makes it very difficult for people and objects to move through space without detection — a profound change in the past two decades for citizens and national security agencies.

3. Ubiquitous technology

Ubiquitous technology is the pervasiveness of technology in individuals’ lives and extent to which they interact with it, knowingly or unknowingly. Phones and computers are so deeply embedded in human lives[46] that it is easy to forget that much of the technology is barely older than teenagers. Google started in 1998. Facebook is 17, YouTube is 16, and the iPhone is merely 15 years old. The way the world produces, handles, and sells data has changed so much in such a short time that regulation, access, and understanding have struggled to keep pace.

Other technologies are equally ubiquitous but less visible, such as the analytics that make sense of data, and the sensors that are omnipresent in the environment. The processes and practices of AI are also pervasive and driven by big data — although often the technical sophistication is pseudo-scientific and makes unsubstantiated claims and assumptions about human behaviour.[47] Big data underpins the future of AI, a term that has become more prominent in the past 10 years and which is often used as a marketing phrase, diluting its meaning and importance.

The technology sector has become increasingly dominated by a small number of companies, which concentrates information flows, critical data sets, and the technical capabilities essential for functioning democracies. A handful of companies have monopolised areas of data abundance, digital connectivity, and ubiquitous technology, creating an “infrastructural core” or ecosystem upon which most other applications and platforms are built.[49] They are therefore able to control global data flows and information services[50] in an unprecedented way. Historical monopolies such as the East India Company, the Vanderbilt empire, Rockefeller’s Standard Oil, and Carnegie Steel wielded similar power. However, contemporary commercial power resides in and through data and information, in addition to its economic weight. This represents a new power base for private enterprise.

According to the former head of the UK’s intelligence and security organisation, these tech companies have “come to know much more about us and our personal habits and tastes than any intelligence agency ever could (or should)”.

Alphabet (Google), Apple, Facebook, Amazon, and Microsoft dominate their market sectors, with unprecedented data stores and analytical capabilities conferring market power.[51] Virtually everyone else — government agencies included — depend on these big tech companies at some level for their infrastructure and information services, particularly cloud computing infrastructure.[52] China has a similar set of firms in Alibaba, Baidu, and Tencent, which are less widely used in the West, [53] although the global appeal of video sharing service TikTok, owned by Chinese company ByteDance, and the regional appeal of instant messaging service WeChat and others may be a sign of things to come. According to the former head of the UK’s intelligence and security organisation, these tech companies have “come to know much more about us and our personal habits and tastes than any intelligence agency ever could (or should)”.[54]

Companies that have monopolised data abundance, digital connectivity, and ubiquitous technology largely control global data flows and analytics, information services, and risks. Since virtually all information is now controlled by or goes through these large technology companies, the security landscape for government has fundamentally changed, creating unprecedented interconnectivity and vulnerability.[55] The Covid-19 pandemic has further accelerated digitalisation in society and contributed to widening power asymmetries between consumers and big tech companies.[56]

While government regulation globally tries to catch up, big tech companies have amassed immense power. In Australia, regulation tends to focus on specific aspects of data abundance, digital connectivity, and ubiquitous technology rather than on the overall landscape. For example, the Australian Competition and Consumer Commission’s Digital Advertising Services Inquiry into anti-competitive behaviour[57] is part of a five-year Digital Platform Services Inquiry (2020–2025), while its News Media Bargaining Code[58] focuses on content distribution.

Identifying and protecting data on citizens may be just as important as protecting other critical infrastructure. Yet big data has shifted the parameters of who creates and owns information about Australians.

The top five big tech American firms have a market value of approximately US$9.5 trillion and earned revenues of around US$1.2 trillion last year.[59] Combined, their market capitalisation is more than five times Australia’s gross domestic product, while their annual earnings are more than twice Australia’s federal government fiscal revenue for 2020.[60] Big data is essential for continued success in the technology market and increasingly across all sectors of society, including the delivery of government services. Big tech’s data dominance is a potent barrier to market entry for others, endowing substantial market power to the largest platforms.[61] This has given the companies control over global data stores, information flows, and services. Consequently, they wield enormous economic and infrastructural power.

Identifying and protecting data on citizens may be just as important as protecting other critical infrastructure. Yet big data has shifted the parameters of who creates and owns information about Australians, and who owns and operates the infrastructure many of our services rely on. The big data landscape is enabling new, non-state actors, primarily tech companies, to deny or change citizen access to services in a way that is at times inconsistent with democratic values and anti-discrimination laws, and which is largely unregulated. For example, algorithmically-driven systems can offer, deny, or mediate access to services or opportunities to people differently, and perpetuate over-monitoring and over-policing of minority groups.[62] Lack of algorithmic, data, and company transparency makes it difficult to fully comprehend and quantify the nature of the threat.

The big data landscape shapes many of the contemporary security challenges Australia faces. Three emerging threats to national security and nation-state power emerge from the big data landscape. First, big data can confer a strategic advantage by enabling information dominance and improving kinetic targeting capability — the application of active military force. Second, big data enables and “democratises” targeting and surveillance. Third, big data drives information warfare as well as social and political interference.

These threats are emerging in the context of a security environment that the Australian government’s 2020 Defence Strategic Update describes as “markedly different from the relatively more benign one of the past”.[63] Confidence in the rules-based global order is being undermined by disruptions from a widening range of sources. Expanding cyber capabilities — and the willingness of some countries and non-state actors to use them — are complicating the strategic environment. Major power competition has intensified and the prospect of high-intensity conflict in the Indo-Pacific, while still unlikely, is less remote. The prevalence of grey zone activities has increased in the Indo-Pacific, involving military and non-military forms of assertiveness and coercion, including rogue cyberattacks targeting Australia’s data infrastructure. The actions are aimed at achieving strategic goals without provoking broad-scale conflict. Such activities have ranged from militarisation of the South China Sea and active interference to big data-enabled information operations, disinformation campaigns, and economic coercion.[64]

Espionage and foreign interference also represent threats to Australia’s way of life. There are more foreign spies and their proxies operating in Australia than there were at the height of the Cold War.

Covid-19 has accelerated the use of digital technologies and pushed more interactions online, significantly adding to the global pool of data[65] and increasing social dependence on technology. Online interactions encourage “echo chambers”, which can polarise society, promote distrust, and create more opportunities for malign interference.[66]

Domestically, Australia faces evolving threats, too. The 2020 Annual Report from the Australian Security Intelligence Organisation (ASIO) notes that the terrorism threat level remains at “probable”, with no prospect it will be lowered in the foreseeable future.[67] Religiously motivated violent extremism remains a significant and real concern.[68] Ideological extremism is more organised, sophisticated, and active.[69] Big data is a force multiplier for all forms of extremism.[70]

Espionage and foreign interference also represent threats to Australia’s way of life. There are more foreign spies and their proxies operating in Australia than there were at the height of the Cold War.[71] Foreign governments are seeking information about Australia’s capabilities, research and technology, and domestic and foreign policy.[72] The big data landscape is changing modern spy craft and expanding the avenues for intelligence collection, as well as offering new ways to identify adversary intelligence operations.[73] The big data landscape expands the techniques available to adversaries to obtain this information and makes them harder to identify and disrupt.[74]

Recognition of the role cybersecurity plays in national security is now much more broadly understood, with a growing raft of measures to address it. However, other aspects of the big data landscape, such as identifying and protecting vulnerable and critical data sets, remain largely unaddressed. The big data landscape is challenging existing models of harms assessment, which have historically been focused on direct physical harm and economic harm.

There are now myriad ways to attack an organisation or individual, with some traditional methods still available only to nation states, but many more available to a wider range of entities. For example, a stock exchange could be impacted in traditional ways by malicious actors, such as through physical damage or destruction, cyberattack, or natural or human-instigated disasters such as flooding or power failure. The big data landscape adds potential new vectors, such as altered or corrupted data, coordinated price manipulation on a mass scale, or information operations resulting in real or perceived insider trading that generates fear in the institutions’ ability to function or lack of trust in its integrity.

1. Big data confers strategic advantage: information dominance is military dominance

Big data can dramatically improve situational awareness and can be weaponised in war to target adversaries. When well integrated into command-and-control systems, information dominance produces military dominance[75] and big data offers global situational awareness on a scale previously not possible. If analysed and processed effectively, big data can be used to outpace an adversary’s decision-making processes. It also offers the ability to challenge or deny an adversary’s situational awareness by poisoning data sets and running adversarial AI systems. It can be used in war to improve targeting for kinetic action.

The military utility of sensing and communications functions and the enhanced potential it offers to target adversaries in conflict is of critical concern.

Big data produced by remote sensors on a large scale can confer strategic advantage by providing situational awareness. It can be integrated into space-based systems and targeting systems to help gain control of territory and territorial approaches, deter traditional military activities, and create uncertainty about the safety of transit in disputed territories. For example, China has deployed a network of remote sensors and communications capabilities in the South China Sea between Hainan Island and the Paracel Islands.[76] Big data capabilities are part of the Blue Ocean Information Network developed by China Electronics Technology Group Corporation (CETC), a state-owned defence company[77] on the US Entity List.[78] Plans for the Blue Ocean Information Network involve expanding the sensor and communications network to the rest of the South China Sea, the East China Sea, and other ocean areas far from Chinese territory.[79]

The military utility of sensing and communications functions and the enhanced potential it offers to target adversaries in conflict is of critical concern. Remote sensors and big data analytics are being combined with traditional Intelligence, Surveillance, and Reconnaissance (ISR) platforms using AI applications as part of the Chinese People’s Liberation Army (PLA) approach to “intelligentised warfare”.[80] The ability to triangulate exact locations based on an array of sensors, especially many that are much cheaper than traditional ISR assets, has particular significance for military transit. As one military commentator noted in relation to military technologies and systems on Chinese-claimed island reefs in the Spratly Islands, “the combined information power capabilities on China’s SCS [South China Sea] outposts…will work synergistically prior to and throughout the military operations to preserve the PLA’s access to information in the SCS battlespace while simultaneously denying an adversary access to information.”[81] In conjunction with traditional military assets, these capabilities provide China with the infrastructure to control large areas, even in international waters.[82]

2. Big data enables targeting, surveillance, and oppression

Big data enables the targeting and surveillance of humans, often in real time. It enables invasive identification of and sometimes complete access to physical and online activity, locations, and movement. The scope for causing individual and societal harm is significant. The volume of personal or personally identifying information available and degree of datafication and connectivity means comprehensive profiles of individuals, interest groups, institutions, political groups, and nation states can be quickly and remotely created. From an individual perspective, this is often expressed as a concern about privacy intrusion, however, from a national security perspective, the big data landscape offers new potential for surveillance and targeting.

The ability to produce a more complete picture of society and individual behaviour creates the potential for a society that is more invasive and repressive of individual autonomy.[83] In the near future, it will become almost impossible to escape digital surveillance[84] — Google’s trackers are present on more than 80 per cent of 1000 popular websites in Australia.[85] The national security implications of near complete data coverage of human lives — which enables tracking, monitoring, and analysis, sometimes in real-time[86] — are underappreciated in policy and public commentary.

An example is the extensive and well-documented surveillance and detainment of Uighurs in Xinjiang, which is enabled by a combination of big data collection and data fusion, among other forms of surveillance.[87] Another example is the high-end spyware Pegasus, designed by Israeli cyber arms firm NSO Group to track terrorists and criminals. It was recently revealed that despite Israeli export controls, Pegasus has been used to spy on journalists, human rights activists, government ministers, diplomats, and businesspeople in democracies and autocratic regimes.[88]

It is already possible to surveil and oppress individuals and groups in more obtrusive, less regulated ways. Censorship of Australians and Australian politicians by WeChat already occurs.

Big data “democratises” the capabilities underpinning targeting and surveillance — functions previously reserved for nation states and their governments. The big data digital footprint and infrastructure used to analyse it is predominately owned by commercial entities, meaning that the data — and ability to derive insight from it — largely resides in the private sector. Much of this collection occurs within companies that monetise their user data and much of it is available for purchase.[89] The scale of data, often accessible in real time, creates uncertainty over when, where, and by whom aggregation, targeting, and surveillance can occur. Where once states exercised surveillance with external authorisation and oversight, big data systems enable tracking, monitoring, and analysis, and make these capabilities accessible to many more actors, in opaque environments with varying, sometimes non-existent, degrees of regulation of their activities.

The pre-conditions already exist for adversaries to purchase or acquire data to hinder states’ ability to achieve national security objectives and harm key elements of democratic societies. It is already possible to surveil and oppress individuals and groups in more obtrusive, less regulated ways. Censorship of Australians and Australian politicians by WeChat already occurs.[90] So does workplace monitoring,[91] private and public space monitoring, and crowd sourcing evidence of crimes,[92] where investigators collect critical information and leads from the public through social media. In future, we can expect the same tools to be used to facilitate espionage, interference, and other methods of interstate competition.

3. Big data drives information and political influence and interference 

Big data can be used to harm individuals and society by providing mechanisms to improve information warfare as well as influencing and interfering in the political and civic discourse that is essential to democracy. The effectiveness of these capabilities is fuelled by datafication and made possible by using the mechanisms of targeted advertising known as “microtargeting”.

Microtargeting, also referred to as personalisation, targeted digital advertising services, customer-targeted advertising, and precision targeting, is the use of data to target small groups or even individuals.[93] The desire to target specific individuals with tailored messages — often advertising — is not new, but the technical capabilities underpinning this are being driven by the big data landscape and enabling contemporary information influence and interference.

To date, targeted advertising has largely been focused on developing new corporate revenue streams. However, it has the capacity to shape an individual’s information environment, including what they see, the choices they are presented with, what they think others believe, and ultimately how they might view the world.[94] Fragmented media landscapes and microtargeting lead to an increase in political and social polarisation.[95] Microtargeting therefore presents significant national security threats. These include mass influence and interference, such as the ability to target specific groups, identify and exploit psychological weaknesses, and interfere in political and civil processes such as elections.

This kind of granular targeting is well established in the electoral and commercial realms[96] and has laid the groundwork for the kind of election influence and interference seen in recent years.[97] Examples include Cambridge Analytica’s role in the 2016 US elections,[98] attempted Russian interference in US elections,[99] and Facebook’s attempts to sell data on 6.4 million young Australians and New Zealanders claiming to identify their emotional state, particularly vulnerability and insecurity.[100] The effectiveness of microtargeting on individual behaviour is inconclusive[101] and unknown outside the big tech companies, which monetise their user data in this way. However, the use of political and commercial microtargeting, combined with a steady stream of big tech whistleblowers,[102] demonstrates the companies themselves are aware of its influence and potential.Big data has exponentially accelerated the process and effectiveness of influence and interference, enabling it to occur at scale and at speed with increased opacity and ambiguity about intent or identification of who is behind the activity. Given the opaque nature of microtargeting processes — and their availability to anyone willing to fund campaigns — they are an ideal mechanism for grey zone information warfare.

Main image: The ubiquitous use of technology enables near complete coverage of human lives across the planet often in real time (Gilles Lambert/Unsplash)