Extraordinary, and increasingly rapid, technological advancements are bringing impressive benefits to our lives. Yet it is equally undeniable that these advances generate new challenges and serious, if not terrifying, threats. In this chapter, I will address the most radical of these threats: the futuristic yet alarmingly near possibility—often evoked with the term “technological singularity”—that AI could achieve autonomy so profound it might harm humanity beyond human control. In the second part of the chapter, I will also explore how an ethical design of artificial systems could be imagined.