In recent years, there has been significant development in the use of machine learning for various tasks involving natural language interpretation. Improving user experience is a top priority for many online services, including search engines and social media platforms. One such service is answering users’ questions. Domain-specific data, such as sacred data, can present unique difficulties for natural language processing because of their intricate structures. In this paper, we employ domain-specific sacred data and a question-and-answer dataset to evaluate and contrast the optimal existing pre-trained deep transformer models for sacred data question answering. Based on the variations of the pre-trained deep transformer models, BERT models, we propose a multilingual domain and task-specific model. To improve the model predictions, we also undertake post-processing and cross-lingual transfer learning steps. On the official test set, our proposed model receives a Partial Reciprocal Rank (pRR) score of 62% recording promising results.