The Bayesian Neural Network (BNN) allows us to learn a probability distribution over network weights. It has potential advantages in medical image registration due to uncertainty estimation. This paper proposes a Bayesian image registration network embedded in a variational autoencoder (VAE) to provide epistemic and aleatoric registration uncertainty at the same time. The posteriors of the weights and latent variables in the VAE are estimated jointly by variational inference. The prior distribution of weights is learned to be the optimal moment-matching prior. Taking the estimated epistemic uncertainty by the BNN, we increase the signal-to-noise ratio with respect to the epistemic uncertainty of the registration model. Extensive experiments on public cardiac image datasets demonstrate that our registration model outperforms state-of-the-art probabilistic registration models on small training sets and cross-domain datasets.