Studies have shown the promising performance of transformer-based models in machine reading comprehension task. However, the performance of these models has not been explored extensively and deeply with Classical Arabic texts. To fill this gap, we first created two datasets: one for the Qur’anic text and the other for the Hadith Sharif, which are considered the main sources of Islam. We chose these two sources because they are more challenging. Their text requires a deep understanding of the content, and their terminology can have an interpretation that is different from that of other books. We then made them available to the research community. Second, we explored the performance of all the pre-trained transformer-based models available for the Arabic language when used as an answering model for the Hadith Sharif questions. Third, we studied the impact of further training the models on a Classical Arabic dataset, such as the Qur’an dataset and/or Modern Standard Arabic datasets. Finally, we selected the best-performing models and applied the ensemble method to their results. The model that achieved the best result was the ensemble of the CAMeL-BERT and CL-AraBERT, with a 0.187 EM, 0.711 F1 score, and 0.631 pRR.