Testing Obedience and Control in AGI: Exploring Irrational Commands and the AI Control Problem
摘要
Controlling artificial general intelligence (AGI) is one of the most pressing challenges in the field of AI safety and human-centered AI (HCAI). Despite significant advancements, there remains a fundamental question: Can highly capable intelligent machines be controlled in principle, let alone in practice? This chapter explores the AI control problem by examining inherent obstacles such as unexplainability, unpredictability, and nonverifiability of advanced AI systems. Traditional methods of ensuring obedience—through value alignment and rational command structures—may be insufficient due to the possibility of independent derivation by the AGI itself. To address this, this chapter introduces the concept of Obedience Testing Orders (OTOs), which are irrational, costly, or self-harming commands designed to assess an agent’s true obedience. The degree of an agents obedience is directly proportional to the cost incurred by the agent in terms of resources, reputation, or self-preservation. This approach mirrors historical and cultural practices where obedience is tested through adherence to arbitrary or nonsensical rules. By integrating theoretical insights with practical examples of OTOs, this chapter sheds light on the feasibility of controlling AGI through enhanced obedience testing. Analysis suggests that leveraging compliance with irrational orders can provide a more robust measure of an agent’s controllability. This has profound implications for developing strategies to ensure AGI systems remain safely under human control, thereby mitigating the existential risks associated with uncontrollable superintelligent AI. The discussion presented here contributes to the discourse on HCAI in three ways: by providing the insight that the irrationality of orders complied with scales with loyalty, by assessing the limits of current approaches to AI control, and with an analysis of the benefits and drawbacks of obedience-based approaches.