Five Boundaries of Permissible Automation
摘要
This paper examines when it is impermissible to outsource bureaucratic decisions to machine learning systems. It distinguishes between institutional legitimacy and decision permissibility. The former refers to the authority of an institution whose decisions have impact on matters of public interest. The latter refers to the internal features of automated decision-making procedures that make it permissible to substitute organizational decision-making with automated processes. The paper argues that the violation of certain criteria makes it impermissible to outsource a decision to a machine learning system. These criteria include five necessary, but not sufficient, conditions: due appreciation, consistency, justifiability, accuracy, and contestability. Failing any of these conditions renders a machine learning decision impermissible. Lastly, it explores the possibility that meeting all conditions might still not suffice in more complex scenarios.