AI's absence from the proposed EU Public Procurement Act -- suggested improvements

Jamillah Knowles & Digit / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/

If we live in the age of artificial intelligence, the proposed EU Public Procurement Act is anachronistic in its omission of AI. AI simply does not feature in the Public Procurement Act — or at best, only as an after-thought or ‘something to think about at a later time’. Given the continuous expectation that procurement will operate as a key governance tool for public sector AI use, that is not good enough.

As is now trite, there are two major interactions between AI and procurement: procurement is the gateway for AI acquisition by the public sector (‘AI procurement’); and AI can be (and is increasingly) used to manage procurement (‘AI for procurement’). Here, we can distinguish the direct use of AI for procurement and the use of AI to monitor procurement (eg to tackle corruption or collusion risks). The latter could be thought about as a meta application of AI for procurement.

Despite the growing understanding of these two/three layers of issues and the emergence of a quickly-expanding academic literature, as well as policy and guidance literature from across a wide variety of jurisdictions, the Public Procurement Act does not include meaningful consideration of AI.

Missing priority

To be fair, other than in relation to some meta applications of AI for procurement, AI also did not feature in the key documents influencing the Commission’s approach to the proposal. The report by the European Court of Auditors and the Council’s conclusions that kick-started the review solely referred to AI in passing (in relation to automated translation by ECA, and about exploitation of public procurement data by the the Council). The Parliament’s resolution on procurement similarly solely stressed that the transition to a digital-first approach would “reinforce the possibility of public scrutiny in the procurement process, especially considering the potential offered by disruptive technologies such as artificial intelligence” and called the Commission to update its tools for monitoring competition in public procurement. As far as I know, only the opinion of the European Committee of the Regions has gone beyond consideration of these meta uses of AI for procurement, and stressed that “it is necessary to provide a specific legal regime for the use of AI in the area of public procurement, thereby guaranteeing a framework of legal certainty” (para 38).

Similar considerations were captured in the public consultation underpinning the proposal. As the Commission’s impact assessment reflects, “several respondents see AI and advanced analytics as a way to improve procurement governance, through automated checks, risk scoring, fraud detection, and carbon-footprint verification. At the same time, they call for clear rules on the use of AI and algorithmic tools in award processes, to maintain transparency, fairness and legal certainty”.

The Public Procurement Act’s approach

This has not translated into any provision in the proposed Public Procurement Act, not even in relation to the (meta) use of AI for procurement. All there is, in the Legislative financial and digital statement (part 4.3), is a consideration that several elements of the envisaged new digital architecture (such as the Commission’s eProcurement platform, the electronic eligibility service, or the procurement data spaces) could interact with the EU AI Act. However, there is a repeated statement that “No AI-based decision-making is foreseen for the core legal obligations. If automated risk detection, ranking, recommendation or fraud analytics are introduced, an AI Act risk/compliance assessment will be carried out.” So, if anything, the proposal suggests that the digital architecture will not include ways to embed AI for the purposes of any of the key aspects of a procurement procedure (ie ‘core elements’). Is this a tenable position? Is this a ‘future-proof’ approach to regulating procurement in the EU, especially on the basis of a single rulebook? What would happen when Member States start (or continue) deploying AI in relation to ‘core obligations’? Would that be permitted? Under which conditions? How would divergence in the (applicable by default) administrative law of the Member States be negotiated?

Regulatory assumptions?

More broadly, it seems that, at best, the Commission operates on the assumption that the EU AI Act would provide the regulatory architecture for the use of AI in procurement (if there was any), but this is only explicit in relation to the meta use of AI for procurement and pending some future impact assessments (not least, because there is no detail whatsoever on what those components will look like in the many delegated and implementing acts required to develop them). Even then, the use of AI for procurement is not really considered in full, such as eg the Public Procurement Act’s omission of the more immediate potential obligation to disclose human-AI interaction under the transparency requirements of Art 50(1) EU AI Act even where no AI-based decision-making takes place—as soon as tenderers or public buyers eg interact with a chatbot.

The procurement of AI is, in itself, not even referred to in the proposal. Perhaps this is based on the Commission’s view that its (meagre) work on this issue to date suffices.

However, this ‘silent approach’ to reflecting the AI/procurement interactions in the Public Procurement Act ignores the fact that, other than transparency, the EU AI Act does not create obligations for the use of AI in procurement and, at best, provides a very weak level of regulatory assurance for AI procurement, and that the model clauses for AI procurement are nothing but an empty shell.

taking (more meaningful) action

All in all, the Public Procurement Act does not contain any provisions on AI procurement, does not fully consider the use of AI for procurement, and only hints at the future consideration of meta uses of AI for procurement monitoring purposes. This is at odds with the pressure for procurement to regulate AI acquisition by the public sector, as well as the pressure to deploy AI to manage procurement processes.

In my view, there are at least three improvements the co-legislators should introduce in the legislative process the Public Procurement Act is undergoing.

First, the Public Procurement Act should reform the EU AI Act and, in particular, Annex III to include the use of AI for public procurement (both directly, and in relation to its meta monitoring) as a ‘high-risk’ AI use case. This would bring procurement within the scope of the EU AI Act, but would still be largely ineffectual due to the exemptions in Article 6(3) EU AI Act. Therefore, the Public Procurement Act should also specify that Article 6(3) EU AI Act is inapplicable to the use of AI for procurement. This would go a long way in addressing the governance of AI use by public buyers.

As a complement—or a rather weak substitute, if modifying the EU AI Act in this way is not (politically) feasible—the Public Procurement Act could include an explicit provision on AI use for procurement to make public buyers directly responsible for the consequences of AI use for procurement, regardless of any responsibility or liability that can be sought from the providers of the used AI solutions—and this could be coupled by an obligation to develop, publish and constantly update a policy on the use of AI for procurement. This should include specific approaches to testing for bias and anticompetitive effects potentially arising from AI use (for discussion, see here).

Second, to mirror this, the use of AI by economic operators participating in procurement procedures would also need regulating. At a minimum, this should be reflected in the rules on exclusion (which need a significant overhaul more generally, in particular to make all exclusion grounds mandatory if the instrument is to remain as a Regulation) and an explicit exclusion ground should cover the misuse of AI in the participation in procurement procedures or in the execution of public contracts.

Finally, AI procurement needs to be given legislative coverage. As there is no chance that a full regime can be included in an Act, and much less that this would be sufficiently dynamic and adaptable, the Public Procurement Act should include a new article mandating the Commission and the Member States, jointly, to develop and constantly update a set of mandatory (not model) clauses for AI procurement. These would need to be a lot more detailed and implementable than the current clauses, and it seems to me that the future clauses need to be jurisdiction-specific (not merely translated into relevant languages) for them to adequately bridge the gap between public and private law rules (including liability) and jurisdictional variations. It would also be important to consider comparative approaches and to reflect on and learn from, eg recent developments in the US with the adoption by the General Services Administration (GSA) of a new clause for incorporation into government contracts of artificial intelligence systems (552.239-7001, Basic Safeguarding of Data within Large Language Model Artificial Intelligence Systems).

These interventions would still need refining and careful drafting and, most likely, would probably still be insufficient to fully assure AI procurement and the use of AI for procurement and its oversight, but they would provide a firm statutory grounding on which to continue the regulatory conversation.

Protecting procurement's AI gatekeeping role in domestic law, and trade agreements? -- re Irion (2022)

© r2hox / Flickr.

The increasing recognition of the role of procurement as AI gatekeeper, or even as AI (pseudo)regulator, is quickly galvanising and leading to proposals to enshrine it in domestic legislation. For example, in the Parliamentary process of the UK’s 2022 Procurement Bill, an interesting amendment has surfaced. The proposal by Lord Clement-Jones would see the introduction of the following clause:

“Procurement principles: automated decision-making and data ethics

In carrying out a procurement, a contracting authority must ensure the safe, sustainable and ethical use of automated or algorithmic decision-making systems and the responsible and ethical use of data.”

The purpose of the clause would be to ensure ‘that the ethical use of automated decision-making and data is taken into account when carrying out a procurement.’ This is an interesting proposal that would put the procuring entity, even if not the future user of the AI (?), in the legally-mandated position of custodian or gatekeeper for trustworthy AI—which, of course, depending on future interpretation could be construed narrowly or expansively (e.g. on whether to limit it to automated decision-making, or extend it to decision-making support algorithmic systems?).

This would go beyond current regulatory approaches in the UK, where this gatekeeping position arises from soft law, such as the 2020 Guidelines for AI procurement. It would probably require significant additional guidance on how this role is to be operationalised, presumably through algorithmic impact assessments and/or other forms of ex ante intervention, such as the imposition of (standard) requirements in the contracts for AI procurement, or even ex ante algorithmic audits.

These requirements would be in line with influential academic proposals [e.g. M Martini, ‘Regulating Algorithms. How to Demystify the Alchemy of Code?’ in M Ebers & S Navas, Algorithms and Law (CUP 2020) 100, 115, and 120-22], as well as largely map onto voluntary compliance with EU AI Act’s requirements for high-risk AI uses (which is the approach also currently followed in the proposal for standard contractual clauses for the procurement of AI by public organisations being developed under the auspices of the European Commission).

One of the key practical considerations for a contracting authority to be able to discharge this gatekeeping role (amongst many others on expertise, time, regulatory capture, etc) is access to source code (also discussed here). Without accessing the source code, the contracting authority can barely understand the workings of the (to be procured) algorithms. Therefore, it is necessary to preserve the possibility of demanding access to source code for all purposes related to the procurement (and future re-procurement) of AI (and other software).

From this perspective, it is interesting to take a look at current developments in the protection of source code at the level of international trade regulation. An interesting paper coming out of the on-going FAccT conference addresses precisely this issue: K Irion, ‘Algorithms Off-limits? If digital trade law restricts access to source code of software then accountability will suffer’ (2022) FAccT proceedings 1561-70.

Irion’s paper provides a good overview of the global efforts to protect source code in the context of trade regulation, maps how free trade agreements are increasingly used to construct an additional layer of protection for software source code (primarily from forced technology transfer), and rightly points at risks of regulatory lock-out or pre-emption depending on the extent to which source code confidentiality is pierced for a range of public interest cases.

What is most interesting for the purposes of our discussion is that source code protection is not absolute, but explicitly deactivated in the context of public procurement in all emerging treaties (ibid, 1564-65). Generally, the treaties either do not prohibit, or have an explicit exception for, source code transfers in the context of commercially negotiated contracts—which can in principle include contracts with the public sector (although the requirement for negotiation could be a hurdle in some scenarios). More clearly, under what can be labelled as the ‘EU approach’, there is an explicit carve-out for ‘the voluntary transfer of or granting of access to source code for instance in the context of government procurement’ (see Article 8.73 EU-Japan EPA; similarly, Article 207 EU–UK TCA; and Article 9 EU-Mexico Agreement in principle). This means that the EU (and other major jurisdictions) are very clear in their (intentional?) approach to preserve the gatekeeping role of procurement by enabling contracting authorities to require access to software source code.

Conversely, the set of exceptions generally emerging in source code protection via trade regulation can be seen as insufficient to ensure high levels of algorithmic governance resulting from general rules imposing ex ante interventions. Indeed, Irion argues that ‘Legislation that mandates conformity assessments, certification schemes or standardized APIs would be inconsistent with the protection of software source code inside trade law’ (ibid, 1564). This is debatable, as a less limiting interpretation of the relevant exceptions seems possible, in particular as they concern disclosure for regulatory examination (with the devil, of course, being in the detail of what is considered a regulatory body and how ex ante interventions are regulated in a particular jurisdiction).

If this stringent understanding of the possibility to mandate regulatory compliance with this being seen as a violation of the general prohibition on source code disclosure for the purposes of its ‘tradability’ in a specific jurisdiction becomes the prevailing interpretation of the relevant FTAs, and regulatory interventions are thus constrained to ex post case-by-case investigations, it is easy to see how the procurement-related exceptions will become an (even more important) conduit for ex ante access to software source code for regulatory purposes, in particular where the AI is to be deployed in the context of public sector activity.

This is thus an interesting area of digital trade regulation to keep an eye on. And, more generally, it will be important to make sure that the AI gatekeeping role assigned to the procurement function is aligned with international obligations resulting from trade liberalisation treaties—which would require a general propagation of the ‘EU approach’ to explicitly carving out procurement-related disclosures.

Public procurement and [AI] source code transparency, a (downstream) competition issue (re C-796/18)

© Diverse Stock Photos/Flickr.

Two years ago, in its Judgment of 28 May 2020 in case C-796/18, Informatikgesellschaft für Software-Entwicklung, EU:C:2020:395 (the ‘ISE case’), the Court of Justice of the European Union (CJEU) answered a preliminary ruling that can have very significant impacts in the artificial intelligence (AI) space, despite it being concerned with ‘old school’ software. More generally, the ISE case set the requirements to ensure that a contracting authority does not artificially distort competition for public contracts concerning (downstream) software services generally, and I argue AI services in particular.

The case risks going unnoticed because it concerned a relatively under-discussed form of self-organisation by the public administration that is exempted from the procurement rules (i.e. public-public cooperation; on that dimension of the case, see W Janssen, ‘Article 12’ in R Caranta and A Sanchez-Graells, European Public Procurement. Commentary on Directive 2014/24/EU (EE 2021) 12.57 and ff). It is thus worth revisiting the case and considering how it squares with regulatory developments concerning the procurement of AI, such as the development of standard clauses under the auspices of the European Commission.

The relevant part of the ISE case

In the ISE case, one of the issues at stake concerned whether a contracting authority would be putting an economic operator (i.e. the software developer) in a position of advantage vis-à-vis its competitors by accepting the transfer of software free of charge from another contracting authority, conditional on undertaking to further develop that software and to share (also free of charge) those developments of the software with the entity from which it had received it.

The argument would be that by simply accepting the software, the receiving contracting authority would be advantaging the software publisher because ‘in practice, the contracts for the adaptation, maintenance and development of the base software are reserved exclusively for the software publisher since its development requires not only the source code for the software but also other knowledge relating to the development of the source code’ (C-796/18, para 73).

This is an important issue because it primarily concerns how to deal with incumbency (and IP) advantages in software-related procurement. The CJEU, in the context of the exemption for public-public cooperation regulated in Article 12 of Directive 2014/24/EU, established that

in order to ensure compliance with the principles of public procurement set out in Article 18 of Directive 2014/24 … first [the collaborating contracting authorities must] have the source code for the … software, second, that, in the event that they organise a public procurement procedure for the maintenance, adaptation or development of that software, those contracting authorities communicate that source code to potential candidates and tenderers and, third, that access to that source code is in itself a sufficient guarantee that economic operators interested in the award of the contract in question are treated in a transparent manner, equally and without discrimination (para 75).

Functionally, in my opinion, there is no reason to limit that three-pronged test to the specific context of public-public cooperation and, in my view, the CJEU position is generalisable as the relevant test to ensure that there is no artificial narrowing of competition in the tendering of software contracts due to incumbency advantage.

Implications of the ISE case

What this means is that, functionally, contracting authorities are under an obligation to ensure that they have access and dissemination rights over the source code, at the very least for the purposes of re-tendering the contract, or tendering ancillary contracts. More generally, they also need to have a sufficient understanding of the software — or technical documentation enabling that knowledge — so that they can share it with potential tenderers and in that manner ensure that competition is not artificially distorted.

All of this is of high relevance and importance in the context of emerging practices of AI procurement. The debates around AI transparency are in large part driven by issues of commercial opacity/protection of business secrets, in particular of the source code, which both makes it difficult to justify the deployment of the AI in the public sector (for, let’s call them, due process and governance reasons demanding explainability) and also to manage its procurement and its propagation within the public sector (e.g. as a result of initiatives such as ‘buy once, use many times’ or collaborative and joint approaches to the procurement of AI, which are seen as strategically significant).

While there is a movement towards requiring source code transparency (e.g. but not necessarily by using open source solutions), this is not at all mainstreamed in policy-making. For example, the pilot UK algorithmic transparency standard does not mention source code. Short of future rules demanding source code transparency, which seem unlikely (see e.g. the approach in the proposed EU AI Act, Art 70), this issue will remain one for contractual regulation and negotiations. And contracts are likely to follow the approach of the general rules.

For example, in the proposal for standard contractual clauses for the procurement of AI by public organisations being developed under the auspices of the European Commission and on the basis of the experience of the City of Amsterdam, access to source code is presented as an optional contractual requirement on transparency (Art 6):

<optional> Without prejudice to Article 4, the obligations referred to in article 6.2 and article 6.3 [on assistance to explain an AI-generated decision] include the source code of the AI System, the technical specifications used in developing the AI System, the Data Sets, technical information on how the Data Sets used in developing the AI System were obtained and edited, information on the method of development used and the development process undertaken, substantiation of the choice for a particular model and its parameters, and information on the performance of the AI System.

For the reasons above, I would argue that a clause such as that one is not at all voluntary, but a basic requirement in the procurement of AI if the contracting authority is to be able to legally discharge its obligations under EU public procurement law going forward. And given the uncertainty on the future development, integration or replacement of AI solutions at the time of procuring them, this seems an unavoidable issue in all cases of AI procurement.

Let’s see if the CJEU is confronted with a similar issue, or the need to ascertain the value of access to data as ‘pecuniary interest’ (which I think, on the basis of a different part of the ISE case, is clearly to be answered in the positive) any time soon.