The EU's Artificial Intelligence Act: Should some applications of AI be beyond the pale?

The Artificial Intelligence Act (AIA), unveiled in April of this year by the European Commission and still making its way through the European Parliament, proposes a “risk-based” approach to the regulation of AI systems, meant to align it with “European values” and make Europe a leader in “trustworthy AI”. The proposed regulation divides AI applications into four different categories based on their estimated degree of risk: those presenting “unacceptable risk”, “high risk”, “limited risk”, and “low or minimal risk”. While applications in the first category are to be prohibited altogether, those in the three subsequent categories are meant to be regulated with gradually lower levels of stringency.

The AIA has already been the subject of much public debate. Perhaps unsurprisingly, it has received criticism from opposing sides: some suggest that the restrictions it imposes might be too stringent, while others worry that it is overly permissive and open to abuse. This article offers some reflections on this issue, with a particular focus on three AI applications categorized as “unacceptable risk”, and therefore singled out for prohibition: 1) systems that would cause harm to users by manipulating their behaviour; 2) state-run “social scoring” of a general nature; and 3) the use of ‘real time’ remote biometric systems for law enforcement.

Defining the scope of “harmful”, manipulative AI applications

Concerns about overly strict restrictions might arise in response to the AIA’s proposed prohibition of AI applications “intended to distort human behaviour, whereby physical or psychological harms are likely to occur", and systems that would "deploy subliminal components individuals cannot perceive or exploit vulnerabilities of children and people due to their age, physical or mental incapacities" (article 16). Such language might be viewed as undesirably ambiguous, potentially covering practices that a liberal society should not want to ban.

Consider for instance that social media companies like Facebook have been reported to be deliberately designing their products to make them addictive to users. Insofar as addiction to a social media app can plausibly be construed as a form of psychological harm, should we conclude that the AIA would require banning Facebook, unless the company were willing to re-design the offending features? Or, as some have asked, would it require banning Instagram, based on its negative impact on the mental health and body image of some teenagers? In light of the curtailment of personal freedoms that they would entail, such implications would seem problematic (although additional age-based restrictions might be more defensible). However, it is unlikely that they accurately reflect the intentions behind this provision of the AIA, which we may instead presume to be targeted at more clearly questionable practices like “dark patterns” (i.e., devious methods used by some websites and apps to trick the user into doing things they may not have wanted to do). Still, greater clarity as to the scope of application of this particular prohibition would be desirable.

Should we fear AI-enabled “social scoring”?

The AIA's ban on "social scoring of natural persons for general purpose by public authorities or on their behalf" (art. 17), another potential development assumed to represent an “unacceptable risk”, has elicited quite different responses so far. Some have characterized it – alongside the recent US-EU joint statement on AI – as largely an attempt to claim the moral high ground vis-à-vis China, while potentially targeting a phantom, based on a misrepresentation of China's social credit system. Yet this proposed ban has also been criticized as insufficient: civil society groups have thus argued that it should be extended to private entities.

It is true that misconceptions about China’s social credit system are relatively common in the West: for instance, in its current form, the system is much more fragmented than is typically assumed, and does not reward or punish every Chinese citizen for their daily behaviour based on a single overall social score. Nevertheless, this does not mean there are no better justifications for wanting to prevent the introduction of “general purpose” social scoring than a demonization of the Chinese system. An allocation of rewards and punishments based on an overall measure of a citizen’s “virtue” or “trustworthiness” does seem to be a natural endpoint of the current Chinese experiment, and it is not inconceivable that it could emerge even in the West. Five years ago, a controversial app named Peeple was launched by a Canadian company, allowing users to rate others (without their prior consent) under three major categories: personal, professional, and romantic. While this is still not as unified as a single overall social score, it was still close enough to generate outrage, causing the app to be watered down.

What, exactly, would be objectionable with such a social scoring system? After all, some might argue that the use of scores as the basis on which to treat individuals more or less favourably is already a common practice even in Western democracies. Examples include credit ratings, commercial loyalty programs, employee performance reviews, and point systems – managed by the state – for driving offences. If such practices are acceptable, why is “general purpose” social scoring of a country’s citizens beyond the pale? One reason is the latter’s greater potential for abuse. If a system’s purpose is to identify generally “good” or “trustworthy” citizens, rather than positive or negative behaviours of a much more specific kind, it seems to offer leaders with authoritarian proclivities a convenient excuse for sneaking inappropriate elements into the assessment scheme. For instance, point deductions for sharing “fake news” – with that category being treated as encompassing all speech the authorities dislike. General assessments of citizen “virtue” also risk encouraging ever more intrusive levels of government surveillance, given the wide range of private behaviours that could be considered relevant to that goal.

Even if safeguards could somehow be imposed to forestall such excesses, a more fundamental problem with the concept of a general social score is arguably its paternalistic and stigmatizing nature. This is clearest in cases where one’s low score would be made public, with the aim of shaming them into “better” behaviour. Yet the concern would still apply under a system that did not disclose a person’s score to any third parties other than the official purveyors of rewards and punishments. At least from a liberal perspective, the sheer existence of such a general score means that the state is sending an unduly patronizing message to individuals about their status as “good” or “bad” citizens, something that is not required to incentivize socially beneficial behaviour. That being said, an emphasis on respect for individual liberty also calls into question the appropriateness of extending a ban on general social scoring to private actors. Presumably, people ought to be free to use apps like Peeple; the key issue is whether we can ensure that they do so in a voluntary and informed manner, without having to suffer unacceptable personal costs should they refuse to be subjected to such social scoring (say, because it had become the norm).

Remote biometric systems for law enforcement: prohibit or regulate?

A third point of contention concerns the AIA’s ban on “the use of ‘real time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement” (art. 18). The ban admits of exceptions in cases of three specific kinds: 1) the search for missing children and other potential victims of crime; 2) the prevention of threats to life or physical safety, such as terrorist attacks; and 3) the identification of suspects for crimes that carry maximum sentences of three years or more. Some, including civil society organizations, have called for the removal of those exceptions, and for the ban to be extended beyond “real time” uses, and to actors other than law enforcement.

The question whether to outright ban, or rather carefully regulate, remote biometric systems such as facial recognition technology, is no doubt a difficult one. The desire to protect citizens’ privacy, and to forestall misuse and wrongful arrests based on potentially inaccurate and biased algorithms, can make a straightforward ban look like the best option. Yet it should also be recognized that a policy of indefinite prohibition would likely involve significant opportunity costs in terms of society’s ability to prevent crime. A substantial public debate is therefore needed to determine under what exact conditions, if any, such technologies can appropriately be used, and what technical solutions might be available to help address the problems of accuracy and bias. A temporary moratorium on their use could be considered while that debate is taking place.

All in all, discussions around the AIA illustrate the challenges involved in seeking to balance human rights and other “European values” against the need to promote technological innovation and its accompanying economic and social benefits in the European Union. While the extent to which EU policymakers must face trade-offs between these different goals is itself a matter of dispute, the general need to balance potentially competing considerations cannot be denied. Whether the AIA gets things just right in this regard remains a matter for further debate, yet at the very least, this effort can be commended for aiming at the right goal.