Joint modeling or multi-task learning by applying speech and text encoders can improve the performance of end-to-end speech translation (E2E ST) with the help of large-scale speech recognition and text translation data in previous work. However, most existing methods require architectural changes of ST training, and the modality gap between speech and text makes it hard to encode both of them with a shared encoder. In this paper, we regard the joint modeling for speech translation and text translation as multilingual NMT modeling, using a single encoder for both speech and text processing. We empirically show that the modality gap can be effectively addressed by modeling modality-aware relative position encoding in the self-attention layer. Experiments on three benchmarks covering 23 languages show that joint modeling over internal/external speech recognition and text translation data, the performance of our single encoder method can lead to significant improvements on multiple ST tasks in both directions (from and to English) compared to the baselines.