Plagiarism is a common issue in academia, affecting the integrity of scholarly work. The rise of Artificial Intelligence (AI) and widespread availability of digital content have made Plagiarism Detection (PD) more challenging, particularly for advanced plagiarism types. Traditional PD tools are often limited in addressing these challenges, especially for non-English languages such as Serbian, due to its morphological richness. This study presents a PD approach and system for Serbian student papers, integrating a Vector Database Management System (VDBMS) with Machine Learning (ML)-based embedding models. The semantic search capabilities of VDBMSs and the contextual representations of embedding models are utilized to improve detection accuracy, particularly for Serbian's linguistic complexities, including its grammatical structure and dual-script usage. The effectiveness of this approach in detecting various forms of plagiarism is demonstrated, highlighting its applicability for non-English, resource-scarce languages.