Passive acoustic recordings can play an important role in biodiversity monitoring and management of human activities, with growing potential due to artificial intelligence. The twaite shad Alosa fallax is a clupeiformes fish found from the northwestern Atlantic to the Mediterranean. It spawns in freshwater and migrates upstream from the sea to rivers in spring. Twaite shad are classed as a threatened species. As part of their spawning behavior, males chase females in circles close to the water surface, producing a distinctive splashing sound that can be used for passive acoustic monitoring. In reintroduction programs, this can be used to measure spawning activity and locate spawning sites. As manual identification of shad splashing events in large audio datasets is time-consuming, an automatic shad splashing detection system is required. Here, embeddings from seven pretrained AI models from different source-domains were used to train a classifier for shad splashing detection. The model BirdNET provided the most informative embeddings. A class imbalance in training data composition was beneficial for classifier performance, and filtering for high-quality positive examples was detrimental. This study shows the value of retraining publicly available, pretrained machine learning algorithms to detect novel challenging sound events against a noisy background.