Cardiovascular diseases are disorders of heart and blood vessels and have become a cause of concern in the recent years. The identification of biomarkers in these diseases will help the clinicians to better manage and prevent the diseases. However, the biomarker identification in cardiovascular diseases dataset becomes challenging owing to limited samples and a large number of features. The conventional feature selection approaches do not perform well owing to the curse of dimensionality. This work proposes an ensemble feature selection approach combining a filter and a wrapper method and finding the common features from the two. The proposed work results in efficient model resulting in an accuracy of 97.01. The results are based on an extensive study employing hyperparameter tuning and focusing on the explainability of the model. It may be noted the proposed pipeline handles variance and bias gracefully. The work finds the important biomarkers for the disease and has the potential of assisting the clinicians in effective and efficient diagnosis of the disease.