Suppose we set some alignment goal — say not physically harming humans. And suppose in some reasonable sense an AI creation pursues that goal (I’ll say a bit about the ‘reasonable sense’ I have in mind in the paper). Still, pretty much any action carries some small risk of harming humans (rather like driving to the store to get a burger carries the small risk of hurting a pedestrian along the way). So even assuming the goal is pursued, some risk threshold will have to be set in context. If it is too draconian, the goal will be paralyzing. If the threshold is set too generously, the result will in time be disastrous. And risk thresholds become particularly important when the downsides are catastrophic (a good case in point: discussions of what risk was tolerable for strangelet particles when the ion colliders were built). And as technology advances, opportunities to take risks of large-scale catastrophes become more plentiful. One problem is that we ourselves are not very good at dealing with tiny probabilities and so it is hard to know what the ideal thresholds are to be. Another is that we tend to rely on ordinary human sensibilities when it comes to setting risk thresholds but such sensibilities cannot really be relied upon when the downsides are catastrophic. My initial way into all this — mainly chosen because it is both interesting and poignant — is to ask how things might go terribly wrong in the world of the Bicentennial Man with a robot that did its absolute best to adhere to Asimov’s ‘Three Laws of Robotics’. I will then move on to discuss some more real-world scenarios — including AI management of various aspects of the economy and weapons systems — to illustrate the importance of risk thresholds.