WORKING PAPER Auditing of Algorithms : Closed-Door Panel of the Florence Competition Summer Conference

On 24 and 25 June 2021, the EUI held, both in Florence and online, the Summer Conference ‘Effective remedies vis-à-vis digital platforms: competition policy and sector regulation at a crossroad’. The event gathered different stakeholders, including officials from well-represented competition authorities, academics, industry representatives and practitioners, to discuss the growing interaction of competition policy and sector regulation with the remedies imposed on digital platforms. We are experiencing many antitrust investigations against online platforms on both sides of the Atlantic and proposed regulatory reforms addressing digital markets issues, such as the EU Digital Markets Act (‘DMA’). This environment provided fertile grounds for a vivid discussion. Different panels focused on the effectiveness of competition law remedies for digital markets, such as data portability, the establishment of data silos, the divestiture of platforms’ assets, interim relief, and algorithms auditing. As an experiment to spur a trusted, free flow of arguments and open conversation between experts, the last-mentioned digital remedies issue, Auditing of Algorithms (‘AoA’), was the object of an invitation-only session under Chatham House Rule. This paper shares the main points that emerged during such a closed-door panel on auditing algorithms, maintaining the anonymity of information sources.

1 This report is a free interpretation of the comments and ideas that emerged at the panel. The report intentionally does not refer to individual comments.

Background: AoA and the Law
In the exciting and challenging time when computer and data sciences enter the legal realm and laws enter the virtual world and all these impact market functioning, the very technical issue of AoA is at the heart of the application of competition law and regulation. Automated software programs such as algorithms and machine learning applications are becoming a general-purpose technology that permeates many market activities by companies across diverse economic sectors. Ubiquitous algorithms process and extract value from data. Firms' pricing decisions, referencing and recommendation systems, content moderation and online advertisement are just a few exemplary business functions that today often rely on automated decision-making algorithms.
Given the strong market impact of algorithms, antitrust authorities, and authorities in charge of consumer protection, telecommunications, or financial ones, increasingly face the problem of understanding how algorithms work to ascertain law compliance or infringement for online conduct and determine further regulatory gaps. Also, the very same firms that employ algorithms with effects on the markets may significantly benefit from a regular AoA. Auditing generally corresponds to assessing an algorithm's impact on variables of interest, such as price or advertisements, based on data collection about the algorithm behaviour within its application context. 1 Besides acquiring an ordinary place in modern competition authorities' enforcement toolbox, AoA has a stronghold in the proposed EU Digital Markets Act, too. 2 Art. 13 of the DMA proposal obliges digital gatekeepers, within six months after their designation, to submit an independently audited description of any techniques used by the gatekeeper to profile consumers for its core platform services and to update such a description at least yearly. Moreover, the proposed Commission's powers to conduct on-site inspections to verify the effective implementation and compliance with the DMA obligations, even through external auditors, include the power to obtain access to and explanations on the gatekeeper's organization, functioning, IT system, algorithms, data-handling and business conducts and to address questions to key personnel (Art. 21 DMA proposal). The recent proposal of the European Commission on the regulation of Artificial Intelligence also contemplates the necessity, in some circumstances, of independent AoA. 3 Overall, AoA practices by the Commission and independent auditors are likely to surge as much as awareness on AoA.

The Departure Point: The Theory of Benefits and AoA Prioritization
We should always acknowledge the 'Theory of Benefits' of algorithms: algorithms can make markets more efficient, on several dimensions, for example, by reducing discrimination or transaction costs as compared with human decisions. 4 Because of the Theory of Benefits, applications of algorithms should not belong to a proscribed list of anticompetitive conducts.
Nonetheless, several theories of harm may emerge from algorithms applications, such as autonomous collusion, 5 non-price personalization, deceptive dark patterns exploiting users' behavioural vulnerabilities and problems in the form of lack of explainability or interpretability. 6 Such theories of harm are not necessarily caused just by algorithms but can arise from other factors like pre-computed scores, labels, different systems in the same pipeline. Considering the pros and cons of algorithms applications and realizing the complexity of today's systems, general AoA obligations are not a panacea to digital market problems and risk doing more harm than good. Instead, mandatory AoA should come after policymakers identify the specific market problems to be addressed and set appropriate measures.
It should also come as no surprise that most AoA uncovers no evidence of unlawful behaviour. For instance, the actual audit of a prominent search engine brought no findings on the problems of filter bubbles and online political polarization. Further, the audit of content moderation systems did not uncover signs of alleged political bias against right-wing users. In the face of negative public perception and lack of trust, the results of AoA often supports digital platforms' claims of abidance to the law. However, AoA may have a preventive role, inducing users to assess their algorithms in anticipation of auditing. Auditing, especially by external independent experts, can also help online platforms to restore their public image.
For competition agencies, AoA is not a discrete activity occurring in a vacuum and just once and for all, but an integral part of their enforcement processes for building evidence on pipeline competition cases or investigations. As a result, aptly formed competition agencies' data units should work side by side with case units helping each other respectively to understand the institutional and technical perspectives and their implications. The seven years of investigations by the European Commission in the Google Shopping case, whose alleged abuse concerns algorithms only tangentially, or the Amazon case pending since 2019, show that checking the properties and functioning of algorithms is far from a trivial effort. 7 Also, even once an infringement is detected and sanctioned, antitrust authorities should monitor compliance with any appropriate remedy, which may require continuous auditing of the algorithms involved.
Without a presumption of harm by algorithms and coping with scarce resources, competition and regulatory authorities should prioritize their auditing scrutiny. A possible prioritization criterion could follow the European Commission 2021 Artificial Intelligence regulation proposal, which regulates AI tools based on their risk assessment. The regulatory intervention is directly proportional to the risks assessed for fundamental rights and safety. Low risk AI applications bear minimal information duties. On the opposite side of the risk spectrum, unacceptable risks lead to prohibiting the AI tools involved. Finally, high-risk AI applications are subject to strict obligations even before market launch because of their significant impact on users' lives, both positively and negatively. Specifically, high-risk AI applications must undergo risk assessment, evaluation of the high-quality datasets fed into the AI tools, traceability check, detailed record-keeping and logging obligations to prove compliance, user notification obligations, human oversight and robustness checks.
Another auditing prioritization criterion could reflect the link between the algorithm application and harm to consumers. For instance, audits of an algorithm that maximizes users' direct spending on games or gambling should perhaps precede those of music recommendation algorithms or adpersonalization ones. Additionally, firm-size thresholds could also ground AoA prioritization, although both big-tech and small firms use algorithms. Regardless of the specific criteria adopted, pre-set AoA prioritization principles provide the legal certainty digital businesses need to plan their research and development investments. Objections to Amazon for the use of non-public independent seller data and opens second investigation into its e-commerce business practices' (European Commission press release, 10 November 2020).

AoA Methodologies
No matter how complex and black-box algorithms are, insofar as they have outputs, they are measurable and auditable. AoA follows different methodologies, all looking for patterns or incongruencies between the algorithm data input and its outcome. To a certain extent, the nascent AoA practice develops upon the established yet creative cybersecurity industry, whose service providers test clients' IT defences through unorthodox methods such as penetration testing, red/blue teams simulation or bug bounty hunters.
Going into algorithms and their outcomes is an interdisciplinary effort where all stakeholders are sharing a collaborative learning curve through any methodology. For a meaningful role in the AoA debate, all stakeholders must develop interdisciplinary teams mixing quantitative and qualitative expertise and composed not just of data scientists but also physicists, engineers, economists, law scholars, sociologists and anthropologists. Universities have a fundamental role in meeting AoA stakeholders' demand for both technologists versed on policy issues and bureaucrats with technical skills. New courses on data ethics must spark regulatory interests in future computer and data scientists, paving the way for young technicians ready and prone for qualitative work. At the same time, there is scope for enriching traditional social sciences curricula so far strange to quantitative rudiments.
In attracting diverse technology competencies, antitrust and other regulatory authorities face the additional hurdle of offering career prospects for professionals that neither have a tradition of public employment nor are used to substantial qualitative work. For instance, lack of technological capacity could be behind the reported European Commission's struggle vis-à-vis Amazon's algorithms since the opening of the antitrust case in July 2019. 8 Nonetheless, a few virtuous examples of dataproficient competition authorities are already leading the way for less than a few years, and their experiences are spilling over the competition community positively. The AoA methodological choice is context-specific and depends on the audit's ventured issues, from specific details of an AI application to its whole system and context. Accordingly, there is no one-size-fits-all approach to AoA. Focusing on the audit timing, AoA can occur ex-ante before an algorithm is put into use or ex-post once it is already operational. In light of the type of access to the algorithm, auditing can be direct thanks to application-programming interfaces that provide interoperability with the algorithm. Because of the involvement of APIs and access to the source code and datasets, direct auditing is either done internally by firms or externally by authorities or independent experts collaborating with the audited firm. Direct AoA comprises so-called sandbox and white box testing. Sandbox testing uses an isolated testing environment enabling users to run programs or execute files without interfering with the run upon application. White box testing refers to the see-through concept and involves testing an algorithm application with detailed inside information of its source code, architecture and configuration.
Alternatively, one could rely on indirect auditing through external soliciting and scraping data from already operating services independently from the audited firm (so-called black-box testing). 9 Indirect auditing applies statistical, data science or reverse-engineering methods upon the scraped big data sets to understand how the services' data inputs relate to the services' outputs. Many of the AoA machine learning tools were developed for in-house use by data scientists rather than for administrative investigations. As a result, the ability to audit algorithms goes hand in hand with their use, and even the algorithms auditing other algorithms might present explainability issues. Indirect AoA often happens as a research activity at universities since it does not require cooperation by the audited firm, whose reactions to auditing results vary significantly.
Any AoA technique stumbles upon matching legal norms and the many underlying economic concepts of competition law with technology practice and designing statistical tests upon them. For example, algorithmic collusion or fairness are challenging notions to associate with computer code. In practice, the matching endeavour requires close interaction between legal and technical professionals and is still undergoing a trial-and-error phase. Ideally, legal norms should be adapted to technical issues to ease the matching. But, more realistically, the law should reflect system complexities through cautious and flexible legal tests that facilitate matching legal norms to technological reality. Instead, the law should not introduce undue rigidity and specificity and force algorithms applications into one-metric legal mechanisms risking foreclosing opportunities for innovation.
Because AoA is a greenfield, there is the scope for collaboration among authorities and firms employing algorithms in converging on voluntary algorithmic standards that ease their auditing by design. However, until such standards or precedents are in place, the novelty of competition law cases involving algorithms calls authorities firms should engage constructively, trying to get the best out of their interactions. For example, investigated firms cannot simply hand out code and petabytes of data to authorities. Instead, they should collaborate in the AoA process with the auditor akin to a technology-transfer scenario with a business partner. At the same time, authorities need humble and flexible approaches, letting opposing technologists first exchange different perspectives on the audited algorithm, leaving legal issues for subsequent confrontation.
It should be avoided the cost of excessive auditing, and one could also consider specific liabilities of auditors. All this may require revisiting the relative weight of type I and type II errors in addressing potential market malfunctioning.

Internal and external Auditing
AoA can occur internally by firms themselves or externally by either authorities or independent auditors. Internal and external AoA obligations are essential complements to competition law enforcement which can pull information from auditing results quickly to initiate and close cases that otherwise suffer from significant information asymmetries.
Internal AoA has the advantages of constituting an early-warning system that can detect problems before product launch and set the stage for subsequent tailor-made external AoA. Irrespective of AoA obligations, large technology firms already do much self-auditing, running countless experiments yearly to improve the quality of algorithm-based services. Furthermore, a first internal audit doublechecked by the authority externally promises as an approach to save the likely limited initial public resources. Though, it requires close collaboration and transparency between companies and authorities. The AI regulation proposal also recognizes that AI sectors are very innovative, and their auditing expertise is constantly growing. Accordingly, for high-risk AI applications such as biometric identification systems that need a third-party ex-ante conformity assessment, the proposed AI regulation accepts that, initially, such an assessment might be performed with internal checks combined with vigorous ex-post enforcement.
External auditing by independent experts should have a prominent role in AoA, especially as consultants of small companies and competition authorities without the resources or in-house competencies to audit algorithms. As recognised by art. 13 of the DMA proposal, the involvement of independent consultants in competition investigations or ongoing monitoring permit antitrust authorities to acquire initial algorithms understanding skills quickly and cost-effectively. Authorities can use their formal investigative powers to obtain firms' data sets and direct access to systems and feed these to independent experts for AoA purposes. However, they cannot rely on external consultants exclusively. In fact, the interactions between technology experts and enforcement officials often span long periods and cross a variety of parallel cases. Therefore, it is in the authorities' interest to build in-house data and technology capacity and reach AoA self-sufficiency soon.
Independent auditors also benefit the audited firms, bestowing the reviewed algorithms muchneeded credibility, especially if involving high-risk applications. Likely, large consultancies will populate the market for external AoA building across various cases their expertise and reputation, although only a handful of small firms has algorithmic auditing skills so far. Like credit-rating agencies in financial markets, the external consultancies may need a degree of profitability at stake with their reputation to deliver high-quality AoA. Yet, the market should remain contestable for high-quality audit results.
Academics may play a relevant role in external auditing as well, upon which competition authorities should not hesitate to draw. Even if differently motivated and without access to in-house data, independent academic auditing can provide helpful information and act with a preventive role. Especially in applications of algorithms that involve mass consumption, algorithms and their market applications should remain accessible for external academic and independent auditing.

Takeaways and Open Issues
In parallel to both external independent experts and firms themselves, competition and other regulatory authorities will be involved ever more in AoA, that is, the examination of algorithms' inputs and outputs looking for patterns or incongruencies. Regulations on automated decision-making algorithms have an essential role in raising awareness of their criticalities, though they should always stress the many benefits they bring to consumers. Because of the 'Theory of Benefits' of algorithms, as confirmed by many audits uncovering no evidence of unlawful behaviour, broad-brush prohibitions against algorithms applications better give way to prioritized AoA obligations addressing aptly identified market problems. Risk assessment approaches such as the one of the proposed EU AI Act are just one of the possible AoA prioritization criteria providing legal certainty to digital businesses.
Several options exist about how to perform AoA based on the timing of the audit (i.e., before or after the launch of the algorithm application) and on the type of access to the algorithm (i.e., direct thanks to APIs or source code versus indirect through solicited or scraped datasets). Still, AoA is always an interdisciplinary effort, mixing quantitative and qualitative skills, where legal norms must match technology practice. All AoA stakeholders share a collaborative learning curve that leaves much leeway to converge on voluntary algorithms and auditing standards. Given its infancy, AoA leaves many relevant issues open. These include the need for coordination both horizontally among national authorities working on AoA from different competencies and internationally among national peers or the questionability of auditing as the only necessary means to detect market harm by algorithms. These and other yet unknown issues will keep algorithms stakeholders busy in years to come.