This paper presents an effective and efficient dual domain network based on Swin Transformer, called SwinDDnet, for Metal Artifact Reduction (MAR) in CT or CBCT images. Challenges in MAR lie in following aspects: first, MAR is very fastidious about pursuing better performance as a clinical application, but the methods grounded in Convolutional Neural Network (CNN) are unable to it due to the locality of CNN; second, although Transformer compensates for the defects of CNN, yet it can not model at various scales and deal with images with high-resolution pixels; third, the diverse morphologies of metal artifacts make it thorny to solve the problem just in the single domain. To handle the issues, we present a new transformer-based approach to overcome metal artifacts in the dual domain. We adopt Swin Transformer as the network backbone, capturing multi-scale context information at a global view and reducing the quadratic computation complexity down to the linear one. Additionally, this network can abate heteromorphic artifacts in the image and projection domains with a concerted effect. Further, we coalesce CT image priors generated by a prior network to refine the quality of final CT images. Finally, extensive experiment results reveal that our strategy outperforms some other methods.