Outlier removal is an essential component of many computer vision tasks. In the task of outlier removal, both convolution and attention mechanisms have demonstrated significant effectiveness independently. However, they are commonly viewed as distinct and alternative methodologies. In this work, we integrate these two methodologies through a specific strategy for the outlier removal task. Specifically, we design a Convolution Self-Attention Fusion (CSAF) block that effectively merges convolution with self-attention to process motion vectors within the motion field, thereby enhancing the subsequent outlier prediction. Extensive experimental outcomes indicate that our method outperforms several existing methods.