Internet of Things (IoT) platforms possess several properties that can potentially jeopardize the safety and security of users, such as distributed deployment, processing sensitive information, and exposed networks. This issue is further exacerbated by the physical nature of IoT allowing devices to compromise the confidentiality and integrity of both persons and property.Some IoT problems can be addressed by taking preventative measures before they happen, which requires making predictions about the future. We design ProvPredictor to gather provenance information in order to train a model to make predictions about potentially unsafe behaviors in the future. To demonstrate the effectiveness of ProvPredictor, we create a realistic deployment using IFTTT, a web-based IoT platform, using the most common IFTTT compatible services and applications in a home environment. We additionally use Agriculture datasets to show how ProvPredictor can operate in industrial systems. We train ProvPredictor on the generated provenance data and find that ProvPredictor can predict violations with over 90% accuracy. With ProvPredictor we demonstrate the advantage that provenance information provides to IoT and the feasibility of a provenance collector that focuses on predicting future behavior.