Predictive coding has recently gained momentum in the machine learning community as an energy based, biologically inspired method to train neural networks via local computations only. While its energy functional has an established interpretation as a variational free energy, its relationship to other variational inference algorithms has so far not been thoroughly explored. In this work, we formally show that the predictive coding energy objective can be re-derived through a careful choice of constraints on a Bethe Free Energy functional. Doing so confers a number of benefits: First, it provides a recipe for deriving closed-form updates for any variable that is part of the predictive coding network, opening the possibility of fully gradient free, Hebbian learning. Second, it allows us to combine the tools of predictive coding with advances from both conventional Bayesian deep learning as well as traditional Bayesian methods. Finally, making the constraints explicit allows them to be manipulated in order to derive new predictive coding based objectives and algorithms. We demonstrate this by introducing a novel predictive coding network employing Matrixnormal-Wishart priors over the weights which can be trained using closed-form updates.