COVID-19 is one of the most contagious diseases in the twenty-first century. This paper examines different computer-aided screening techniques for COVID-19 detection from chest X-ray images. In this study, hybrid methods employing transfer learning, fusion of deep features and/or handcrafted features with shallow machine learning classifiers are used for COVID-19 detection. Eight pre-trained convolutional neural network (CNN) models such as AlexNet, MobileNetV2, DenseNet201, InceptionV3, ResNet50, EfficientNetB0, VGG16, and VGG19 are considered. Three handcrafted feature extraction models such as Histogram of Oriented Gradients (HOG), Scale-Invariant Feature Transform (SIFT) and Local Binary Pattern (LBP) are employed. Four shallow machine learning classifiers such as decision tree, K-nearest neighbors (KNN), random forest and support vector machine (SVM) are employed. To improve the classification performance, various fusion strategies are used that combine handcrafted features with deep features. This paper focuses primarily on complex fusions such as HOG with ResNet50 and EfficientNet, HOG with ResNet50, LBP with ResNet50, LBP with EfficientNet, and HOG with EfficientNet, VGG16 with VGG19, LBP with VGG19, LBP with VGG16, VGG19 with HOG, VGG16 with HOG, and large-scale fusions combining different models and feature sets. The results of the experiments show that feature fusion greatly increases COVID-19 detection accuracy. Notably, surpassing other models and feature combinations, the fusion of HOG + ResNet50 + EfficientNet and LBP + ResNet50 + EfficientNet with SVM attained the best accuracy of 98.4%.