A strange Scottish case on evaluation of tenders -- Boston Sci. Ltd v Common Service Agency [2016] CSOH 132

I find the recent Scottish case Boston Scientific Limited v The Common Service Agency [2016] CSOH 132 most confusing. This is a case of healthcare-related procurement whereby the Scottish NHS' central purchasing body, the Common Service Agency, was tendering framework contracts for the supply of certain types of medical equipment--for simplicity, pacemakers and implantable defibrillators.

The litigation concerned the applicable award criteria and the ensuing evaluation of the tender submitted by Boston Scientific Limited. Even if the case seems to be decided mainly on procedural grounds (the claimant, or pursuer in Scottish terminology, seemed to have been time-barred in raising a challenge against the published award criteria), it raises substantive issues that, in my view, should have been dealt with differently by the Court.

The tender had been advertised and the relevant invitation to tender (ITT) had published the applicable award criteria. For each of the lots in which the framework agreement was to be divided, the tenders would be assessed against a pass/fail criterion of essential features (ie mandatory technical specifications) and then evaluated on a 60:20:20 split of the maximum score of 100. The offered price would carry a 60% weight, whereas two quality criteria would carry 20% each: (a) the inclusion of certain defined desirable features and (b) the longevity of the devices.

The challenge was based on Boston Scientific's submission that the assessment of the longevity of the pacemakers must have been wrong. In a nutshell, Boston Scientific claimed that their position as market leaders and the existence of independent tests that demonstrated that their devices had a very long individual life led them to the conclusion that 'there must have been a failure to compare like with like because if there had not been such a failure [Boston Scientific] would have had the highest longevity scores' [para 15]. In short, the tenderer was not convinced that its competitors could (truthfully) have offered devices with a superior longevity.

Boston Scientific's submission of improper technical evaluation of the tenders is complicated by two additional factors. First, that tenderers had only been asked to declare (or self-certify) the longevity of their devices without providing any supporting evidence. Second, that the criteria applicable to the evaluation of the longevity component were not all that clear.

Self-certification of verifiable technical characteristics?

On the first issue, the complaint considers that the contracting authority was not allowed to include as award criteria elements based on pure self-declaration and which it intended not to verify. Indeed, the case seems peculiar because the Common Service Agency 'had stated clearly prior to the date for submission of tenders that supporting evidence was not sought.  It had protected itself in a different way by making clear that the framework agreement would include a clawback provision if battery life fell short of the figure submitted in a tender' [para 18].

This seems to me to be a peculiar way of conducting business because the longevity of devices that need to be implanted in the human body seems a rather important technical characteristic (as submitted by Boston Scientific, but dismissed by Lord Tyre in his Opinion, despite the relevance of this issue for the purposes of EU consumer law as discussed here), and the abrogation of the power to check compliance with technical specifications in this regard seems odd, regardless of the inclusion of financial penalties in the contract. The Judgment relies on two English precedents that would support the legality of relying on self-certification of compliance with contractual terms. Most importantly, it ignores the EU precedent in EVN and Wienstrom (C-448/01, EU:C:2003:651), to which one of the English cases refers, though. A reference to EVN paras [50]-[51] would have sufficed to quash the award procedure (I am thankful to Karen Wontner and Erik Plas for having raised this point in private correspondence).

First, Lord Tyre relies on Public Interest Lawyers v Legal Services Commission [2012] EWHC 3277 (Admin), Cranston J at para [64] to justify the acceptability of self-certification. However, in my view, this precedent is inapplicable here. First, because it concerned an on-going requirement to be discharged during the execution of the contract (ie an element closer to a contract performance clause than a technical requirement) but, most importantly, because in the previous paragraph of that speech Cranston J stressed that 

... the principle behind its decision was the need to ensure the equal treatment of tenderers through the objective and uniform application of the criteria in their assessment. The principle applies whether or not the public authority is able to verify the criteria. If it is able but omits to do so, that is as much a breach of the duty as if it sets criteria which cannot be verified. That is because the outcome may be an inequality of treatment of tenderers through the equal treatment of unequals, i.e. the equal treatment of those meeting and those failing to meet the tender requirements. After all, it is trite law that equality of treatment means not only treating like cases alike but unlike cases differently [at 63].

And this led Cranston J [at para 65] to insist on the need for robust verification where the contracting authority relies on self-certification by the tenderers. This is important in the context of the Boston Scientific v Common Service Agency dispute because, this case, 'Although it was accepted that in some cases a contracting authority might have a duty to validate information provided by a tenderer, this was not such a case. The defender did not have the means to verify independently the figures for longevity provided by tenderers' [para 18]. The issue here would have been whether this inability of independent verification (a) covered a complete lack of engagement with existing technical information and (b) was not attributable to the contracting authority itself and its decisions on how to organise the procurement procedure. Generally, one would expect that the entity running framework contracts for medical supplies has (or has access to) necessary technical knowledge in any case. Thus, this point of the case remains obscure and, in my opinion, shows excessive deference to the contracting authority.

Second, Lord Tyre relies on Parker Rhodes Hickmotts Solicitors v Legal Services Commission [2011] EWHC 1323 (Admin), McCombe J at paras [35]-[40], which in turn refers back to Public Interest Lawyers v Legal Services Commission. The difficulty with this second case is that its ratio rests on the construction or interpretation of the tender documentation, rather than an assessment of the requirements of the principle of non-discrimination of tenderers--which was the legal basis for the challenge in Boston Scientific. Importantly, in Parker Rhodes, the relevant part of the Judgment focuses on the fact that the Information for Applicants (IFA) document had not indicated how the contracting authority would proceed to verifying specific aspects of the offers, which the Court considered to cover the possibility of relying on self-certification.

To me, this makes both precedents irrelevant for (if not contradictory to) the assessment of the claims raised by Boston Scientific, which aimed to strike down the procurement process on the basis that the contracting authority had appended a significant weight to a criterion it actually decided not to verify at all. In my view, there are good arguments under the principle of good administration (Art 41 CFR) to demand that contracting authorities only evaluate what they can assess and, even more, that they do not claim not to be in a position to assess technical characteristics of the products they are buying--if nothing else, by reliance on the rules on test reports, certification and other means of proof (now under Art 44 Dir 2014/24/EU). 

Longevity, price, both or none of the above?

Additionally, and focusing on the point of the need to construct or interpret the tender documents as published, the second argument raised by Boston Scientific deserves attention as well because, indeed, the criteria applicable to the evaluation of the longevity component were not all that clear. In that regard, it must be noted that the ITT had established that:

In relation to longevity, the tender receiving the highest total longevity score would receive 20 points.  Each other tender would receive “20‑X points where X = 0.2 x the percentage by which each price in each tender exceeded the lowest price tender achieving the lowest total price score” (Boston Scientific v Common Service Agency, para [5], emphasis added).

This seems odd because the criterion that is aimed at scoring longevity is (or, at least, seems to be) referential to the price of all tenders except that of the tender with (self-certified) longer individual device life. In my view, this is a breach of the general scoring rule included in the ITT, according to which price would carry a weight of 60%. This would not be true except for the tender self-certifying highest longevity, and all other offers' price would be taken into account twice (once for the price component itself, and a second time for the scoring of longevity). This is, simply, technically incorrect and, in my view, should have sufficed to cancel the tender.

However, this does not seem to be the whole story and a mistake must have happened in the preparation of the ITT (there seems to be an obvious explanation if one thinks in terms of copy and paste ...) because, in a debriefing letter, the contracting authority had indicated to Boston Scientific that:

The weighting for Longevity was 20% therefore in each lot the longest longevity submitted received 20 points. The scoring guidance in section 3.3 clearly identifies the points allocated to longevity and how the tender would be scored ... (i.e. if one product had longevity or 100 months (longest) it would score 20 points and if a different product submitted had a longevity of 50 months it would score 10 points)  (Boston Scientific v Common Service Agency, para [10], emphasis added).

Now, this is the natural understanding of a relative scoring for longevity, but it happens not to be the scoring rule disclosed in the ITT, which made reference to relative prices rather than relative longevity. Such a substantial deviation between disclosed scoring rule (even if absurd) and its application seems to run against the basic requirements of the principles of transparency and equal treatment, as recently recast by the Court of Justice of the European Union in TNS Dimarso (for a comment, see here).

In my view, this should also have been taken into account by the Court and, rather than dismissing the challenge, Lord Tyre should have sought to understand better whether the longevity criterion had been assessed as the debriefing letter said, or rather as the ITT established (which could have led to abnormal results ultimately preventing Boston Scientific from making much sense of the scores obtained). Most likely, a divergence between the published scoring rules and the actual evaluation of the tender should have led to a cancellation of the award in any case.

Overall, I think that there are two main problems with the Judgment in Boston Scientific v Common Service Agency, and both of them seem to me to result from a lack of engagement with the case law of the Court of Justice of the European Union by the Scottish court. First, because it is truly abnormal to allow for self-certification of a technical requirement that can be assessed and verified by the contracting authority--at least, by reference to technical documents. Second, because it is also truly remarkable that a contracting authority can evaluate tenders in a way that deviates from the published criteria without the reviewing court picking up on this important aspect (or anomaly) of the process. Ultimately, in my opinion, this is a strange case. But also a very technically deficient Judgment and an incorrect decision.

CJEU opens door to manipulation of evaluations and fails to provide useful guidance on the use of 'soft quality metrics' in the award of public contracts (C-6/15)

In its Judgment of 14 July 2016 in TNS Dimarso, C-6/15, EU:C:2016:555, the Court of Justice of the European Union (CJEU) issued some important clarifications on the requirements applicable to the disclosure of evaluation methods under the EU public procurement rules. However, it also turned down the opportunity of clarifying what are the limits of the discretion that contracting authorities enjoy when deciding which evaluation methods to use and, more importantly, it failed to address the important and quite specific concerns about the use of 'soft quality metrics' that AG Mengozzi had raised in his Opinion in this case (as discussed here, where background to the case is offered).

The case broadly raised two main legal issues. First, whether in addition to the disclosure of the award criteria and their weighting (as required by Art 53(2) Dir 2004/18 and now Art 67(5) Dir 2014/24), contracting authorities must also disclose in the tender documentation, or at some point prior to the review of the offers, the evaluation methods they plan to use in the assessment of the tenders. Second, whether having disclosed a numerical weighting applicable to the quality and price criteria (50/100 each), the contracting authority was right to assess the quality criterion in accordance with a soft qualitative ‘high — satisfactory — low’ scale, not referred to in the contract documents.

no obligation to disclose (or indeed establish) evaluation rules prior to the review of the tenders

Regarding the first issue, after reiterating its case law on the purpose of the rules on disclosure of award criteria and their weighting, and stressing their relevance in ensuring equality of treatment between tenderers both when they formulate their tenders and when those tenders are being assessed by the contracting authority (para 22), the CJEU adopted a position that I find surprising. The CJEU stressed that

it is possible for a contracting authority to determine, after expiry of the time limit for submitting tenders, weighting factors for the sub-criteria which correspond in essence to the criteria previously brought to the tenderers’ attention, provided that three conditions are met, namely that that subsequent determination, first, does not alter the criteria for the award of the contract set out in the tender specifications or contract notice; secondly, does not contain elements which, if they had been known at the time the tenders were prepared, could have affected their preparation; and, thirdly, was not adopted on the basis of matters likely to give rise to discrimination against one of the tenderers (see judgment of 21 July 2011 in Evropaïki Dynamiki v EMSA, C‑252/10 P, not published, EU:C:2011:512, paragraph 33 and the case-law cited) (C-6/15, para 26). 

However, it did not apply this reasoning by analogy to evaluation methods as could have been expected. On the contrary, the CJEU adopted a very lenient approach and, after confirming that neither the rules in the Directive nor the previous case law referred to an obligation to disclose evaluation methods, it went on to establish that

29 ... an evaluation committee must be able to have some leeway in carrying out its task and, thus, it may, without amending the contract award criteria set out in the tender specifications or the contract notice, structure its own work of examining and analysing the submitted tenders (see judgment of 21 July 2011 in Evropaïki Dynamiki v EMSA, C‑252/10 P, not published, EU:C:2011:512, paragraph 35).
30 That leeway is also justified by practical considerations. The contracting authority must be able to adapt the method of evaluation that it will apply in order to assess and rank the tenders in accordance with the circumstances of the case.
31 In accordance with the principles governing the award of contracts provided for in Article 2 of Directive 2004/18 and in order to avoid any risk of favouritism, the method of evaluation applied by the contracting authority in order to specifically evaluate and rank the tenders cannot, in principle, be determined after the opening of the tenders by the contracting authority. However, in the event that the determination of that method is not possible for demonstrable reasons before the opening of the tenders, as noted by the Belgian Government, the contracting authority cannot be criticised for having established it only after that authority, or its evaluation committee, reviewed the content of the tenders.
32 In any event, pursuant to the principles governing the award of contracts ... the determination by the contracting authority of the method of evaluation after the publication of the contract notice or the tender specifications cannot have the effect of altering the award criteria or their relative weighting (C-6/15, paras 29-32, emphasis added). 

The reasoning of the CJEU raises two surprising issues, in my view. First, the CJEU seems to conflate the need for the contracting authority to establish an evaluation method that is adapted to the particularities of a given tender (which seems correct, para 30) with the need for the contracting authority to be able to do that at any time (which seems incorrect, para 31). Accepting that the contracting authority can design ad hoc evaluation methods for each of the contracts it tenders does not imply that it can leave this important aspect of the evaluation process for a late stage. Logically, it would seem that setting the award criteria, their weighting and establishing the rules according to which they will be evaluated are different aspects of one same decision: how will the tenders be evaluated so that the contracting authority can decide which one is the most economically advantageous?

It does not seem diligent for the contracting authority to set out the award criteria and their weighting without having determined the way these will be applied in the evaluation. It also seems to create unnecessary uncertainty to tenderers. This is very clear in relation to the use of automatic formulae in electronic auctions, which need to be disclosed to the tenderers prior to their use (Art 54(5) Dir 2004/18 and Art 35(6) Dir 2014/24).  There does not seem to be a good reason for these considerations not to apply to the use of evaluation methods and to require that the contracting authority is diligent in setting them up in a timely manner (ie when it is setting out the award criteria and their weighting).

Second, and more surprisingly, the CJEU fails to extend to the evaluation method the most obvious and minimal guarantee to avoid (impossible to prove) discrimination, ie determining the illegality of establishing (evaluation) criteria relevant for the assessment of the tenders after the evaluation committee has reviewed them (para 31). Before anything else, it must be noted that the CJEU accepts that 'the method of evaluation applied by the contracting authority in order to specifically evaluate and rank the tenders cannot, in principle, be determined after the opening of the tenders by the contracting authority'. The reasoning should not have been as a matter of principle, but as a point of absolute requirement.

However, it is not clear why the CJEU concedes that 'in the event that the determination of that method is not possible for demonstrable reasons before the opening of the tenders, as noted by the Belgian Government, the contracting authority cannot be criticised for having established it only after that authority, or its evaluation committee, reviewed the content of the tenders'. There is no indication whatsoever in the Judgment of which reasons may have been adduced by the Belgian Government to try to justify the impossibility of establishing the evaluation method before having reviewed the tenders. This is amazing because it makes it impossible to understand where the threshold of impossibility lies and, more importantly, because there do not seem to be any good reasons to accept that a diligent evaluation committee can be allowed to decide on the evaluation method after it has already seen the content of the tenders. Whether this is done in a presential meeting or remotely, there is no justification for the assessors not to agree on the evaluation rules first (and document them), and then proceed to the evaluation. In my view, the CJEU has neglected the need to ensure the right to good administration and, in particular, the need to ensure the most basic guarantees that tenderers are treated impartially and fairly, and that relevant matters are dealt with in a timely fashion (as required by Article 41(1) Charted of Fundamental Rights).

The final protection that the CJEU tries to (re)establish in the case by stressing that 'the determination by the contracting authority of the method of evaluation after the publication of the contract notice or the tender specifications cannot have the effect of altering the award criteria or their relative weighting' is inane and insufficient because the possibility of establishing and playing with evaluation rules after having seen the content of the tenders leaves way too much scope to coming up with rules that allow for an ex post rationalisation of the choice of a given winning in tender without necessarily violating the pre-disclosed information on the applicable award criteria and weightings. This deserves stern criticism.

the use of 'soft quality metrics' in the evaluation of tenders

Moving on to the second issue concerning the use of 'soft quality metrics', such as the ‘high — satisfactory — low’ scale in the case at issue, in my opinion, the CJEU also carried out a defective analysis. The shortcomings of the analysis derive from the fact that the CJEU uses the answer to this second aspect to try to compensate for the weakness of its answer to the first question. Indeed, the CJEU premises the analysis of the use of the 'soft quality metrics' on the assessment of whether their use altered the applicable award criteria and their relative weighting. The reasoning of the CJEU is as follows:

35 It appears that that procedure did not make it possible to reflect, when ranking the tenderers in order to identify the most economically advantageous tender, differences in the quality of their tenders relative to their price, while taking account of the relative weighting of the award criteria resulting from the indication ‘(50/100)’. In particular, it appears that that procedure was capable of affecting the price criterion by giving it decisive weight relative to the tenders ranked in the [‘high — satisfactory — low’] scale of quality ... It is for the referring court to ascertain whether the relative weighting of each of the award criteria published in the contract notice was in fact complied with by the contracting authority during the evaluation of the tenders.
36 While the contracting authority may use a scale for the evaluation of one of the award criteria without it being published in the call for tenders or the tender specifications, that scale may not, however ... have the effect of altering the relative weighting of the award criteria published in those documents (C-6/15, paras 35-36, emphasis added). 

I find this problematic because it does not address the core problem of using 'soft quality metrics' at all. Should they have been disclosed to tenderers or, more importantly, should they have been published in the tender documentation together with an explanation of why specific weightings could not be established, the use of this scale would be unobjectionable under the TNS Dimarso test, while still allowing for very subjective and difficult to objectively verify comparisons of the tenders. This leaves the question of which evaluation rules are compatible with the two main requirements in the evaluation of tenders--ie that the award rules, not only the award criteria themselves and their weighting, have to '(i) to be linked to the subject matter of the contract (ie, to be ‘relevant’), and (ii) to allow the contracting authority actually to determine which tender is economically the most advantageous (ie, to be ‘enabling’)' [A Sanchez-Graells, Public Procurement and the EU Competition Rules, 2nd edn (Oxford, Hart, 2015) 380]. By failing to clarify that 'soft quality metrics' are not enabling and do not provide sufficient objectivity to the evaluation process so as to make sure that the contracting authority does not overstep the limits of its discretion, in my view, the CJEU has left too much space for manipulation in the assessment of tenders.

This is something I had criticised [A Sanchez-Graells, Public Procurement and the EU Competition Rules, 2nd edn (Oxford, Hart, 2015) 388], even if in relation to the award criteria (but the arguments apply equally to evaluation rules meant to assess them] considering the following:

Restrictions Derived from the Inclusion of Non-Quantifiable or Subjective Award Criteria, and the Ensuing Need to Objectify Treatment of Qualitative Criteria. Another way in which the selection and weighting of award criteria could give rise to distortions of competition—and, probably, to discrimination amongst tenderers—would be through the introduction of non-quantifiable criteria, or essentially qualitative or subjective criteria that significantly diminished the possibilities of an overall objective appraisal of the tenders or conferred on contracting authorities unrestricted freedom of choice amongst tenderers. In this regard, even if article 67(2)(a) of Directive 2014/24 allows for the taking into consideration of this type of criterion—referring, in general terms, to criteria such as ‘technical merit’ or ‘aesthetic characteristics’—the requirements of relevance and enabling character of the award criteria (see above, this section), as well as the need to avoid conferring on contracting authorities unrestricted freedom of choice and to ensure that the award criteria make provision for an objective assessment of tenderers, should be taken into particular consideration and constrain the decisions adopted by the public buyer.[1]
As regards the requirement of relevance of such qualitative award criteria, it should be stressed that the circumstances under which considerations such as aesthetic characteristics or technical merit will be relevant and material to the subject-matter of the contract are relatively limited (at least if they are unrelated to performance or functional requirements, which are quantifiable and, hence, do not generate significant difficulties). Moreover, it is submitted that they will generally be associated with tenders that should be ruled by the requirements applicable to design contests—which are specifically regulated and set special rules in this respect (see arts 78 to 82 of dir 2014/24),[2] particularly aimed at ensuring the objectivity and independence of the members of the committee entrusted with the evaluation of qualitative or subjective elements of the proposals. Consequently, aesthetic characteristics or technical merit might be assigned very limited relevance in other types of tendering procedures. The substantial irrelevance of such qualitative or non-quantifiable aspects will, then, require only limited consideration in the majority of the cases, if at all.
Moreover, in order to ensure transparency and impartiality, contracting authorities should (as far as possible) set objective or quantifiable proxies to measure primarily subjective or qualitative characteristics of the tenders; or, at least, set up mechanisms (possibly based on the rules regarding design contests) to ensure an impartial appraisal of subjective or qualitative dimensions of the tenders. If such quantification, or ‘proximisation’ or approximation, is possible, the possibilities for discrimination or distortion of competition will be smaller. Consequently, the adoption of this requirement seems desirable whenever its implementation is feasible.
Therefore, a restrictive approach towards the permissibility of the use of these criteria as the basis for the award of contracts—again, in cases other than design contests—seems appropriate. Consequently, this type of consideration should remain as a secondary criterion, or as a rather marginal complement, to objective and easily quantifiable criteria used to determine the award of the contract to the most economically advantageous tender. Along these lines, and attending to the subject-matter of the contract, contracting authorities should give proper weighting to qualitative or subjective criteria (even if ‘quantified’)—which, in our opinion, should be rather limited and marginal in most instances.
To sum up, it is submitted that contracting authorities are bound to ensure the objective and transparent assessment of tenders, particularly by i) avoiding undue recourse to qualitative or non-quantifiable (subjective) award criteria in procedures other than design contests, and ii) assigning them a proper (limited) weighting; and, in general, they are under a duty to exercise self-restraint in their decisions regarding such criteria, particularly when failure to do so could result in their exercise of unrestricted freedom of choice amongst tenderers and/or generate distortions of competition or discrimination of tenderers.
[1] See: S Arrowsmith, The Law of Public and Utilities Procurement. Regulation in the EU and the UK, Vol. 1, 3rd edn (London, Sweet & Maxwell, 2014) 766–71.
[2] See S Arrowsmith, The Law of Public and Utilities Procurement, 2nd edn (London, Sweet & Maxwell, 2005) 829–39;  PA Trepte, Public Procurement in the EU: A Practitioner’s Guide, 2nd edn (Oxford, Oxford University Press, 2007) 232–4; and C Bovis, EC Public Procurement: Case Law and Regulation (Oxford, Oxford University Press, 2006) 248–51.

Should evaluation committees Be Banned From Using 'soft quality metrics' when they assess Public tenders? (C-6/15)

In his Opinion of 10 March 2016 in TNS Dimarso, C-6/15, EU:C:2016:160, Advocate General Mengozzi has addressed the general question whether EU public procurement rules 'read in the light of the principles of equal treatment and transparency, [require] that a contracting authority should always, or, if not, in certain circumstances, make known in advance, in the contract notice or the contract documents, the method of evaluation or weighting rules used to assess tenderers’ bids'. The case is to be decided by the Court of Justice of the European Union (CJEU) on the basis of the phasing-out rules in Art 53(2) of Directive 2004/18, but the interpretation will be relevant for the future application of Art 67(5) of Directive 2014/24.

In my view, the case is interesting, not primarily because of the discussion on whether evaluation methods need to be disclosed together with award criteria and their weightings, but more importantly because it brings to light the simple fact that some evaluation methods are unable to meet the requirements of the EU rules--to the effect that the award phase needs to enable the contracting authority to actually determine which is the most economically advantageous tender with a sufficient degree of precision and certainty. Thus, I critically assess AG Mengozzi's excellent opinion from this perspective.

The Dimarso case

In this case, a Belgian contracting authority issued a call for tenders for the provision of services and indicated that the award criteria would be as follows:

1 Quality of the tender (50/100)
Quality of the preparation, organisation and execution of the work on the ground, and of the encryption and initial data processing. The services proposed must be described in as much detail as possible. It must be clear from the tender that the tenderer is capable of taking on the whole contract (minimum 7 000 samples / maximum 10 000 samples) within the prescribed 12-month delivery deadline.
2 Price (50/100)
Cost of delivering the contract in relation to the basic sample (7 000 samples) and cost per additional batch of 500 addresses supplied (amounts inclusive of VAT).

There was no further indication of how these criteria would be applied. When it came to evaluation of the tenders received, the evaluation team 'evaluated and compared with each other on the basis of the criteria set out above. First, the tenders were examined and evaluated on the basis of the “quality” criterion. For this, each tender was unanimously assigned a given score (high — satisfactory — low). Then, the price criterion was applied. On the basis of those scores, a final ranking was established' (Opinion in C-6/15, para 5, emphasis added).

Dimarso submitted a bid that scored high on quality grounds and was the highest on price. The contract was awarded to a competing tenderer which offer also scored high on quality and was lower in price. Dimarso challenges the way the evaluation team applied the award criteria on the following grounds:

the evaluation committee appears to have evaluated the tenders on the basis of the ‘high — satisfactory — low’ scale, not referred to in the contract documents, in relation to the tender quality criterion, whereas, according to Dimarso, it is clear from the contract documents that a score of 0 to 50 points should have been allocated to each tender. As regards the price criterion, the evaluation committee also failed to carry out an adequate examination, comparison and final assessment of the tenders taking into account the award criteria as set out in the contract documents, including the “50/100” weighting given to each of the award criteria in the call for tenders (Opinion in C-6/15, para 8, emphasis added).

This question raises then two issues: (1) whether the evaluation committee could rely on 'soft metrics' in order to apply the quality award criterion; and (2) whether such 'soft metrics' could be combined with straightforward price comparisons. I find these two questions of great practical relevance, so it is worth looking closely at AG Mengozzi's reasoning on these issues.

Assessment under Art 53(2) Dir 2004/18

It is worth reminding that Art 53(2) Dir 2004/18 established that

[when the award is made to the tender most economically advantageous from the point of view of the contracting authority], the contracting authority shall specify in the contract notice or in the contract documents ... the relative weighting which it gives to each of the criteria chosen to determine the most economically advantageous tender.
Those weightings can be expressed by providing for a range with an appropriate maximum spread.
Where, in the opinion of the contracting authority, weighting is not possible for demonstrable reasons, the contracting authority shall indicate in the contract notice or contract documents or, in the case of a competitive dialogue, in the descriptive document, the criteria in descending order of importance.

At this point, it is worth stressing that the only difference between Art 53(2) Dir 2004/18 and Art 67(5) Dir 2014/24 is that, in relation to the third paragraph, the seemingly permissive drafting of Art 53(2)III Dir 2004/18 ('Where, in the opinion of the contracting authority, weighting is not possible for demonstrable reasons') is tightened up in Art 67(5)III Dir 2014/24 ('Where weighting is not possible for objective reasons'). Given the strict interpretation that AG Mengozzi proposes for Art 53(2) Dir 2004/18 (which is to be shared), his Opinion will be equally relevant for the future interpretation of Art 67(5) Dir 2014/24 [along the same lines, see A Sanchez-Graells, Public Procurement and the EU Competition Rules, 2nd edn (Oxford, Hart, 2015) 384-385].

Going back to the Dimarso case, AG Mengozzi starts by summing up the content of this provision by stressing that

the obligation to indicate not only the award criteria but also ... the relative weighting given to each of those criteria, except where there are good reasons why weighting is not possible, at the time of publication of the contract notice or contract documents ... serves to fulfil the requirement of compliance with the principle of equal treatment and the associated obligation of transparency (Opinion in C-6/15, para 20).

And that

contracting authorities have an obligation to indicate the weightings of the award criteria in the contract notice or the contract documents. It is only in the event that this proves impossible, for demonstrable reasons, that those entities may opt to prioritise those criteria, which prioritisation must in any event be adequately disclosed in the contract notice or the contract documents (Opinion in C-6/15, para 23, emphasis added).

The AG clarifies (paras 24-28) that the dispute in the case at hand is not whether having indicated that Quality (50/100) and Price (50/100) meant that both award criteria had equal weight or how they had to be combined amongst themselves to reach a final ranking of tenders, but that it is rather

in essence, [whether] the method of evaluation used (‘low — medium — high’) was so vague that it prompted the contracting authority to downgrade the assessment of the ‘quality’ criterion relative to that of the ‘price’ criterion, since the second criterion alone was actually capable of eliminating three of the four tenders submitted. In reality, therefore, Dimarso contends, the price criterion benefited from a higher relative weighting than the 50% previously announced in the contract documents. In other words, Dimarso submits that, if the method of evaluation had been made known to tenderers in advance, at the stage when the contract documents were published, it would inevitably have had an effect on the preparation of the tenders (Opinion in C-6/15, para 29, emphasis added).

AG Mengozzi then approaches this argument in stages. His reasoning heavily rests on two aspects. First, that it is clear that Art 53(2) Dir 2004/18 does not explicitly impose an obligation to disclose the evaluation method in addition to disclosure of award criteria and their weightings (para 32). Second, and notwithstanding that literal interpretation of Art 53(2) Dir 2004/18, that the CJEU has been clear in the imposition of restrictions on the way the evaluation team carries out its tasks (paras 37 ff). In my reading, the bone of his argument is as follows.

In relation to the setting of sub-weightings (or weighting factors for award sub-criteria), the CJEU has indicated that this is not a breach of EU procurement rules provided three conditions are met: ie '[1] that it does not alter the criteria for the award of the contract set out in the contract documents or the contract notice, [2] that it does not contain elements which, if they had been known at the time the tenders were prepared, could have affected that preparation, and [3] that it was not adopted taking into account matters likely to give rise to discrimination against one of the tenderers' [with reference to judgments in ATI EAC e Viaggi di Maio and Others (C‑331/04, EU:C:2005:718, paragraph 32); Lianakis and Others (C‑532/06, EU:C:2008:40, paragraph 43); and Evropaïki Dynamiki v EMSA (C‑252/10 P, EU:C:2011:512, paragraph 33); Opinion in C-6/15, para 40, emphasis added].

The same restrictions should be applicable to the adoption of an evaluation method because it can create the same effects as the adoption of sub-weightings--or, in his words, 'it is not inconceivable that a method of evaluation may have an effect not so much on the award criteria themselves as on the weighting of those criteria and, as such, may contain elements which would have been capable of influencing the preparation of tenders if that method had been made known to tenderers in advance... In that event, the ex post determination of such a method for evaluating tenders by a contracting authority would be unlawful and should, therefore, have been disclosed in advance in the contract notice or the contract documents' (Opinion in C-6/15, para 46). Therefore, 'the lawfulness of a method for evaluating tenders which is determined by a contracting authority ex post depends on whether the three conditions established by the Court’s case-law ... are met' (Opinion in C-6/15, para 47).

© iStockphoto.com/RichVintage

© iStockphoto.com/RichVintage

Opportunity for further clarification

Having disposed of the core of the case, AG Mengozzi goes on to suggest that the CJEU take this opportunity to clarify its case law and to stress that the adoption of evaluation methods need to be subjected to tighter requirements. His arguments are based on the use of 'soft quality metrics'--and, more specifically, on the distrust in their ability to actually enable the contracting authority to identify the most economically advantageous tender (MEAT)--as it emerges from his explanation of his main concern:

 

let us imagine that, of the tenders submitted, one was far superior, in terms of quality, to the other three, including those that were rated ‘high’. In other words, one of those tenders could have been ranked ‘excellent’ in the assessment of the ‘quality’ criterion. The price proposed by that tenderer would then have reflected the excellence of the quality of the services proposed by it and would therefore in all probability have been higher than the prices offered by the other tenderers. However, since ‘excellent’ did not feature on the range of scores (low — satisfactory — high) chosen by the evaluation committee, that tender of excellent quality could not but be rated ‘high’, at the very most, in relation to the ‘quality’ criterion. Since the price proposed by the tenderer of that bid was higher than those proposed by the others, possibly even by some tens or hundreds of euros, that bid had to be rejected... in that situation, ... the contracting authority might have been deprived of the tender representing the best value for money, contrary to the spirit in which the selection of tenderers on the basis of the most economically advantageous tender takes place (Opinion in C-6/15, paras 56-57, emphasis added).

AG Mengozzi considers that this is an unsatisfactory state of affairs and, in my reading, proposes that the existing case law of the CJEU is clarified so that contracting authorities do not create a situation where tenderers submit offers which positive attributes are not captured by the evaluation method. His proposal thus focuses on the need to disclose the evaluation method to be used from the start of the procurement process. In his view, 'the likelihood is ... that, if the method for evaluating tenders in the light of the ‘quality’ criterion, as established by the contracting authority, had been known in advance by the potential tenderers, it would have been capable of affecting the preparation of their tenders' (Opinion in C-6/15, para 60). Therefore,

the contracting authority (to which it will fall to ensure that the tendering procedure benefits from maximum legal certainty and to protect itself against actions for the annulment of that procedure) must determine the method or methods to be used to evaluate tenders in the light of the award criteria as early as possible. It would be reasonable to suggest, then, that, if that is the case, there does not appear to be any overriding reason such as to justify a refusal by the contracting authority to make known to potential tenderers the methods of evaluation in question, which it will in any event already have had to determine before the call for tenders (Opinion in C-6/15, para 63, emphasis added).

AG Mengozzi then goes on to discuss whether the condition should be to only require upfront disclosure of evaluation methods which have the potential to create a substantial impact effect on the preparation of the tenders, which he dismisses (paras 70 ff), on the basis that the system would be properly balanced 'by the obligation incumbent on the unsuccessful tenderer, which bears the burden of proof, to demonstrate, by reference to specific examples in its legal action, the differences (substantive as well as purely formal) which its tender would have exhibited if the elements of the method of evaluation in question or the method itself, which the contracting authority neglected to communicate, had been adequately disclosed before the tenders were prepared' (Opinion in C-6/15, para 73). 

personal critique

I share AG Mengozzi's views and concern, but I think that his proposal simply to disclose evaluation methods upfront would only carry us half way in sorting out the unresolved issue of the use of of 'soft quality metrics' in the evaluation of tenders. Regardless of upfront disclosure, which needs to take place, a method for the evaluation of quality aspects of procurement tenders that classifies tenders in pre-determined, tight 'quality levels' is bound to offer sub-optimal results. In the extreme,

a binary approach—ie, an approach based on meeting or not meeting a criterion, or an ‘all-or-nothing’ (or zero/one) approach—seems less desirable than a gradual approach or the adoption of sliding-scale-based evaluation rules ... whenever possible, it seems preferable that contracting authorities evaluate the degree to which tenders comply with each of the specified award criteria on a sliding scale (such as granting them points from 0 to 10, or 1 to 5, or any other scale). In this regard, the weighting of criteria will become less harsh and the appraisal of the tenders will arguably reflect with greater accuracy their relative strengths and weaknesses according to the overall set of award criteria' [A Sanchez-Graells, Public Procurement and the EU Competition Rules, 2nd edn (Oxford, Hart, 2015) 391; Along the same lines, but opting for a monetary equivalent approach, see MA Bergman and S Lundberg, ‘Tender Evaluation and Supplier Selection Methods in Public Procurement’ (2013) 19(2) Journal of Purchasing and Supply Management 73]. 

More importantly, the assessment of quality elements needs to take place in a manner that does result in a loss of information of the relative quality of the offers. It has been the settled case law of the CJEU that 

although [the EU rules do] not set out an exhaustive list of the criteria which may be chosen by the contracting authorities, and therefore leaves it open to the authorities awarding contracts to select the criteria on which they propose to base their award of the contract, their choice is nevertheless limited to criteria aimed at identifying the tender which is economically the most advantageous (Case C-532/06 Lianakis [2008] ECR I-251 29 (emphasis added); Case 31/87 Beentjes [1988] ECR 4635 19; Case C-19/00 SIAC Construction [2001] ECR I-7725 35–36; Case C-513/99 Concordia Bus Finland [2002] ECR I-7213 54 and 59; and Case C-315/01 GAT [2003] ECR I-6351 63–64. See also Case C-448/01 EVN and
Wienstrom
[2003] ECR I-14527 37).

In my view, it is particularly relevant 'to stress the need for award criteria (i) to be linked to the subject matter of the contract (ie, to be ‘relevant’), and (ii) to allow the contracting authority actually to determine which tender is economically the most advantageous (ie, to be ‘enabling’)' [A Sanchez-Graells, Public Procurement and the EU Competition Rules, 2nd edn (Oxford, Hart, 2015) 380]. The same reasoning must apply (functionally) to the selection of evaluation methods (for the reasons explained by AG Mengozzi, ie that they create the same effects). 

Quite frankly, in view of the clear example AG Mengozzi has given us (para 56), I would have no doubt that the use of 'soft quality metrics' is not enabling because it does not allow the contracting authority to identify, with an adequate level of precision and certainty, the most economically advantageous offer. Ultimately, thus, they should be banned as a matter of EU law--and, more generally, of good procurement practice. I do not expect the CJEU to go as far as to agree with this, but I think it would be the only consistent solution, and one that would do away with the problem, rather than trying to fix it simply with the remedy of more transparency--which seems to be the token fix-all solution in procurement law.