The development of human speech begins in early childhood. It is very important to identify a child’s speech disorders in a timely manner. Because speech impediments can impact a child’s learning and social interactions. Classification of speech defects at an early age will allow correction and treatment to be carried out in the most effective way. Analysis of the literature showed that CNN models can successfully detect speech defects. Therefore, this paper explores the architecture of a convolutional neural network to solve the problem of speech defect detection. To use a CNN, audio signals of a child’s speech are first converted into spectrograms (images representing the frequency and time characteristics of the audio signal). The convolved neural network layers are then used to automatically extract essential features from the spectrogram. The article describes the architecture of the CNN for identifying four types of speech defects (dyslexia, stuttering, dysphonia, or dyslalia), which made it possible to obtain recognition of defects with recognition results of 77–79%.