Cardiovascular diseases stand as the leading cause of death globally. While many irregular heartbeats are not severe, certain rhythm disorders can increase the risk of stroke or heart failure. Therefore, distinguishing between various heartbeats is important for identifying heart disorders. Recognising them accurately from electrocardiograms is challenging even for skilled doctors, and patients may not always understand that they have heart issues. This research aims to identify an ensemble of transformer models for classifying specific arrhythmia heartbeats from electrocardiogram signals since it is difficult to correctly distinguish supra-ventricular premature heartbeats from normal heartbeats. The ensemble incorporates three deep learning models with modified architectures, using different class mappings for dataset transformations. The model’s performance is evaluated by comparing recall metrics with a baseline convolutional neural network model and a transformer model trained with an under-sampled dataset for predicting three heartbeat classes. The analysis shows that the ensemble model achieved higher recall metric values for supra-ventricular premature heartbeats and ventricular premature contractions while reaching high overall and per-class metrics.