Maturity indicators: what framework is needed for their use?

There are many indices designed to assess the sensitivity or maturity of states with regard to cybersecurity issues. This multiplicity reflects the protean nature of these same risks, which are of interest to states at several levels. An indicator such as the National Cyber Power Index (NCPI), produced by the Belfer Center of the Harvard Kennedy School is thus based on the notion of power, and reduces its field of study to the most developed countries in the sector to produce a global hierarchy of cyber powers. To evaluate the ‘cyber power’ of the states studied, this index is based on a series of indicators measuring the capabilities and intentions of the countries studied. These are derived from 7 more general criteria corresponding to the objectives pursued by the states that use cyber means. Among these 7 objectives are the surveillance of local attacker groups, the collection of information for national security purposes, and the destruction of an adversary’s infrastructure.

However, even within indicators constructed according to the same paradigm of “power” which structures the NCPI, evaluations can change according to the variables and criteria considered. For example, the report entitled Cyber Capabilities and National Power: A Net Assessment of the International Institute for Strategic Studies (IISS) includes in its 7 comparison criteria the notion of dependence[1], which is less present in the NCPI. And while the same nations are ultimately at the top of the hierarchy, the assessments may differ from one ranking to another for certain countries such as France, whose desire for independence and ability to potentially nationalise its network cores is, for example, noted by the IISS index.

Broaden the perspective for more immediate use

The perspective is broader for one of the major cybersecurity development indices, the International Telecommunication Union’s Global Cybersecurity Index (GCI). Unlike the above, it takes into account all 193 UN member countries as well as the State of Palestine in its assessment of the measures taken by states to improve their resilience to cyber risks. However, while the focus here is more on “civilian” aspects and on the level of commitment of administrations to improve their cybersecurity, there is a complementarity with the indices presented above that measure “power” through the use of common criteria such as the development of a national cybersecurity strategy or the participation in the production of responsible behaviour standards.

The GCI also illustrates the practical use that can be made of this work. In its third and latest edition in 2020, the International Telecommunication Union notes the feedback it has received from governments on the use of its Index. The Global Cybersecurity Index is used by some governments to improve coordination (and awareness) between the various actors in the sector, but also to compare national measures with the best practices of regional neighbours, and more generally to collect comments and feedback on the initiatives taken.

Indicators with well understood limits

While the indices provide a slightly more detailed look at the cybersecurity capabilities of states, they are still limited—even within the scope of the research undertaken by the institute that created them—by a lack of comprehensiveness that the institute acknowledges. For example, the Belfer Center notes that, despite its importance, it cannot currently include the variable “amassing wealth and mining cryptocurrencies” [2]in its assessment of cyber power, due to insufficient data. The International Telecommunication Union, which—unlike the Belfer Center or the IISS—is dependent on responses to questionnaires submitted to states, recalls that several times, it has had to conduct its own research to compensate for the lack of data, since the data available “could not properly reflect the cybersecurity posture of a country.”[3]

But while the “hard” limitations of indices can be overcome by using multiple indices in a complementary analysis, a less easily grasped challenge is the ever-changing nature of threats. As Klara Jordan, Chief Public Officer at the Cyber Peace Institute, reminds us: “When you think about indices, it’s usually a measure for cybersecurity. I have often found that these indices are based on a static threat. Developing these indices takes a long time, but during that time the threats evolve. The threat is not static, while our indices are.” It thus seems that the next major advance in the field could be found in the development of more reactive monitoring tools, allowing for the rapid identification and formalisation of new cyber challenges facing the various players in the field. At the heart of this is the crucial issue of data collection. A new axis of reflection for institutions?