The citizen is not a dataset

Abstract

AI-informed decision-making is increasingly creating inroads into highly consequential public domains. Intellectual property and secrecy considerations should not without more trump democratic public values considerations in such critical domains. Reliance on proprietary models entails that algorithm set-up, methodology, and functioning will remain undisclosed to the public bodies, including to those adversely affected by decisions informed by it and/or to citizens, removing any opportunity for scrutiny and meaningful democratic oversight. In a context where the users are public authorities – increasingly operating and using high-risk systems in areas hugely sensitive for citizens and their lives – it is urgent and salient to rethink democratic accountability in this specific context. Is emerging legislation, like the EU AI Act, and other regulator y initiatives elsewhere adequately engaging with end users as (democratic) citizens? How are public values adequately protected in proprietary models? What more is required of private companies engaging with the public interest?