The World Bank has recently published its report Benchmarking Public Procurement 2017, where it presents a 'cross-country analysis in 180 economies on issues affecting how private sector does business with the government. The report covers two thematic pillars: the procurement process and complaint review mechanisms'.
The information is structured around eight main indicators, which cover the following areas:
- Needs assessment, call for tender, and bid preparation: The indicators assess the quality, adequacy, and transparency of the information provided by the procuring entity to prospective bidders.
- Bid submission phase: The indicators examine the requirements that suppliers must meet in order to bid effectively and avoid having their bid rejected.
- Bid opening, evaluation, and contract award phase: The indicators measure the extent to which the regulatory framework and procedures provide a fair and transparent bid opening and evaluation process, as well as whether, once the best bid has been identified, the contract is awarded transparently and the losing bidders are informed of the procuring entity’s decision.
- Content and management of the procurement contract: The indicators focus on several aspects during the contract execution phase related to the modification and termination of the procurement contract, and the procedure for accepting the completion of works.
- Performance guarantee: The indicators examine the existence and requirements of the performance guarantee.
- Payment of suppliers: The indicators focus on the time and procedure needed for suppliers to receive payment during the contract execution phase.
- Complaints submitted to the first-tier review body: The indicators explore the process and characteristics of filing a complaint before the first-tier review body.
- Complaints submitted to the second-tier review body: The indicators assess whether the complaining party can appeal a decision before a second-tier review body and, if so, the cost and time spent and characteristics for such a review.
The report aims to make progress in the much needed collection of more information, particularly of statistical nature, about the procurement systems that exist around the world. In its own words, '[i]t aims to promote evidence-based decision making by governments and to build evidence in areas where few empirical data have been presented so far. As researchers recognize, “the comparison of different forms of regulation and quantitative measurement of the impact of regulatory changes on procurement performance of public entities will help reduce the costs of reform and identify and disseminate best practices.”' [with reference to Yakovlev, Tkachenko, Demidova & Balaeva, 'The Impacts of Different Regulatory Regimes on the Effectiveness of Public Procurement' (2015) 38 (11) International Journal of Public Administration 796-814].
The report also recognises some of its main substantive and methodological limitations (see p.26). However, even taking those into account, the benchmarking exercise seems rather imperfect and with limited potential to inform policy-making and reform. A couple of examples will illustrate why.
First, in terms of the methodology for the scoring of procurement systems, I am not sure I understand the logic for the award of points or the scale used to weight the different criteria. For instance, when assessing the accessibility of the procurement process, procurement systems are awarded 1 point if bidders are required to register on a government registry of suppliers, and 0 points if there is no registration requirement. To my mind, this is contrary to what logic would dictate because a system that does not require previous or additional registration is more open than one that does.
Similarly, when assessing the existence and requirements for the provision of bid securities, procurement systems get get a score of 1 for either option they provide in a range of questions concerning whether a bid security or a bid declaration is required, whether the bid security amount is no more than a certain percentage of the contract value or value of the submitted bid, or no more than a certain flat amount; whether suppliers have choice regarding the form of bid security instrument; or if bidders are required to post a bid security instrument, whether there is a time frame for the procuring entity to return the instrument. Additionally, procurement systems are awarded an additional maximum of 1 point for each of the forms of bid security instrument they accept: cash deposit, bank guarantee, insurance guarantee (1/3 of a point each). This means that systems that have more flexibility in the way they regulate bid securities will get higher scores (which is fair enough), but that systems that do not require bid securities will get no points. This, for instance, makes the UK (50 points) lag behind Spain (94 points) in this indicator, despite the fact that the UK is recorded as having no bid security requirement and Spain being recorded as requiring a bid security proportionate to the value of the contract (I am not assessing this information which, at least in the case of Spain, requires some nuances). once again, this is contrary to what logic would dictate because procurement systems that do not require bid securities are more open and accessible (particularly to SMEs).
Second, in terms of the comparisons that can be made with the scores as published, I am not sure that the way the information is presented can actually help understand the drivers of different scores for different countries. Most points are awarded on the basis of a yes/no answer to given questions. Given that some questions are rather open-ended or simply confusing (eg the question concerning Criteria for bid evaluation queries whether the procurement system includes "Price and other qualitative elements", but all procurement systems get a score of 1 regardless of the answer), their ability to allow for comparisons is minimal. Moreover, the individual scoring for each criterion is not provided, which prevents direct comparisons even where questions are narrower and actually award different scores to different answers.
Overall, sadly, I am afraid that the report Benchmarking Public Procurement 2017 can only be seen as a first step towards creating a useful system and scoring matrix to benchmark all public procurement systems in the world. I would think that this is possible, particularly once the field work of information collection is in place (unless it was collected as direct responses to the questionnaire linked to the scoring rule) and that the published version of the report can be significantly improved solely on the basis of a better analysis of the raw information collected by the World Bank team. On that point, it is a shame that this information is not published by the World Bank and I would invite them to reconsider the possibility of publishing the database of raw information, so that more specific proposals on how to improve the scoring method without having to collect additional information can be developed.