Despite their success for numerous medical image analysis tasks the use of convolutional neural networks and vision transformers (ViT) have an important limitation of permutation- and rotation-dependency. By design the convolutional filter and positional encoding in ViTs impacts the performance with oftentimes severe robustness issues when using scans with a different input orientation. For biomedical images of tissue samples the orientation is arbitrary and ultrasound scans can be flipped depending on usage. The successful rototranslation equivariant networks tackle the problems at the expense of probing multiple versions of the same filter within each layer, resulting in many times higher computational demand.We revisit and expand this concept by implementing it into MetaFormers that are already partially equivariant and have very few layers which require extra computations. Our experimental validation on several MedMNIST datasets demonstrate the advantages of rotational- and permutationinvariance for the replicated Roto-ResNet18 and our novel Roto-MetaFormer-S12.