In end-to-end speech translation (E2E ST), multi-task learning is often applied due to the scarcity of labeled ST data. However, the modality gap between speech and source text poses a significant challenge for transferring knowledge from machine translation (MT) models to speech translation models. Currently, one of the main approaches to address this problem focuses on utilizing a powerful self-supervised pretrained speech encoder to learn the text embeddings from the MT model. This method involves cross-modal representation learning and heavily relies on the capability of the speech encoder. However, pretraining such a speech encoder is expensive and difficult to replicate. To this end, we propose the Progressive Dual Encoding (PDE) method, which aims to provide a lightweight, pretraining-free approach for E2E ST encoding. This is achieved by introducing an additional text encoder that collaborates with the speech encoder to progressively explore a shared representation space under a novel multi-scale constraint. In this way, our method allows the dual-modality encoders to naturally output similar representations, thereby avoiding the challenges of cross-modal representation learning and alleviating the burden on the speech encoder. We evaluate our method on the MuST-C dataset. The experimental results demonstrate that our method achieves competitive performance with other methods while using fewer parameters in the speech encoder. Moreover, it surpasses other methods by 0.6–1.8 average BLEU score when the number of parameters is increased.