Cardiac auscultation is the simplest and first-line method for detecting cardiac disorders. Congenital Heart Disease (CHD) detection in paediatric patients using the auscultation method is particularly challenging due to the presence of innocent murmurs. Hence, echocardiography is used as the diagnosis-confirming tool. Unfortunately, its accessibility is limited in many primary health centres. To address the issue, the work proposes a deep learning-based model for detecting CHD from phonocardiogram (PCG) signals or heart sounds. The model focuses on the multi-class classification of paediatric heart sounds, including the most predominant two CHDs, atrial septal and ventricular septal defects. A real-time self-collected dataset recorded by the cardiologists in clinical conditions is employed in the work. The work experimented with audio-trained transfer learning models and developed a modified VGGish-based transformer model for the 5-class classification of paediatric PCGs. Depth-wise separable convolution and Light Attention Connected Modules helped to reduce the complexity and number of parameters of the proposed model. The model achieved 93.4% accuracy, 94% precision and 91.4% Kappa and Matthews Correlation Coefficient. The transferability of the model is validated using the public multi-class GitHub dataset. The results prove that the proposed model aids in the early detection of CHD and the initiation of prompt treatment.