Generative Artificial Intelligence (Gen-AI) tools can support both teachers for preparing and delivering lessons and students for solving exercises across various subjects in education. However, in relation to mathematics education some Gen-AI tools may be more appropriate whereas others may not be able to cope efficiently with a specific level of mathematics. In this paper, a pilot qualitative and quantitative performance evaluation of mathematics-oriented Gen-AI models, such as MathosAI, Photomath, MATHia, Wolfram Alpha, ChatGPT Pro, and Symbolab are presented. The comparison is based on performance evaluation using both qualitative and quantitative analysis where the qualitative assessment is derived by analysing user reviews, views derived from the literature, and the impressions of the authors after using those tools. Quantitative assessment is based on the ability of the aforementioned tools to solve typical exercises from the first-year secondary school mathematics curriculum. The results of the qualitative evaluation showed that different tools have different strengths and weaknesses, hence the choice depends on the users’ status, their requirements and the purpose for which they want to use the application. The results of the preliminary quantitative evaluation showed that MathosAI and MathGPT Pro outperformed other tools in exercises from easy to difficult level. However, while these tools are useful in mathematics education, they often provide inaccurate solutions. Therefore, students and teachers should evaluate the answers generated by these Gen-AI tools and not consider them absolutely correct.