Detecting obstacles on railway tracks, such as rocks, is crucial for train safety. In this paper we propose a two-shot architecture for rocks detection: semantic segmentation is used to identify track regions, and a patch extractor is employed to guide a multi-expert system combining the decisions of a convolutional neural network (CNN) classifier and of a Vision Language Model (VLM). The former offers rock detection capability learned from the domain-specific training set, while the latter, pre-trained on millions of general image-text tuples, can recognize rocks and related concepts and distinguish them from similar object categories; these features make them complementary tools that can enhance each other’s performance by combining precise expertise with adaptive generalization. As the experiments confirm, the proposed approach achieves 0.897 F1-score, outperforming the CNN classifier of 5 percentage points and the VLM of 7 percentage points, demonstrating a notable reliability in rocks detection on railway tracks.