Workplace safety, during interaction with automated machinery and robots, can be improved by monitoring both robot and human actions to reduce accidents and injury. Crucial to safety is a low latency detection system. Here we propose to decrease latency by detecting the intention of an operator to press a fault button of an industrial robot, so that it is stopped before the fault button is physically pressed. To do so, we exploit an extremely low latency, vision sensor - i.e. event cameras - to detect fast motion in the fault-button area. We prove that our system can initiate a shutdown up to 500 ms before the physical button is pressed. We analyse the parameters of the algorithm with respect to three different industrial scenarios and environments, and propose an adaptive threshold algorithm to reduce tuning requirements. An interactive application is provided to easily deploy the proposed visual fault-button detection.