In this paper, we introduce an innovative approach to weakly supervised medical image segmentation with box annotations. Different from the previous methods which simply utilize a single conventional network with the same augmentation techniques widely used in supervised segmentation, we aim to introduce diverse augmentations and heterogenous networks to leverage the box annotations for promising generalization ability. Specifically, to amplify the diversity between the contents within the box and its surroundings, we propose the interior and exterior box augmentation (IEBA) technique, in which distinct augmentation techniques are employed for regions inside and outside the bounding boxes. Also, for the purpose of selecting pseudo-labels of superior quality, we propose the pseudo-label filter module (PLFM) to eliminate unreliable pseudo-labels. Besides, as CNN demonstrates superior capabilities in acquiring local information, and ViT specializes in capturing global context, we facilitate a bidirectional learning process between CNN and ViT through quadruple cross consistency losses (QCCL). In inference, we only employ the superior model from the validation set to obtain parameter efficiency. Our approach is evaluated across four tasks on two public datasets, utilizing the 3D dice similarity coefficient as the evaluation metric. The experimental results show that the proposed method outperforms the state-of-the-art comparison methods.