This study explores the use of artificial intelligence (AI) chatbots in mathematics education. With the increasing integration of AI tools such as ChatGPT in higher education, particularly in engineering courses, this research investigates whether students can identify errors in AI-generated solutions. Data was collected from over 200 s-semester engineering students at Gdańsk University of Technology, who evaluated mathematical problems solved by three different AI chatbots, assessing the correctness, difficulty, and usefulness of the AI solutions for self-study. The results revealed that students consistently rated AI-generated solutions higher than those provided by human teachers, with significant differences in evaluations. This suggests a potential challenge in students’ ability to critically evaluate AI responses, possibly due to overreliance on the technology. Additionally, a high school-level mathematics task from the Polish Matura exam was analyzed as part of the study, providing further insight into AI’s performance across different educational levels. The findings highlight the importance of fostering critical thinking skills in students, ensuring they can effectively integrate AI into their learning while maintaining the ability to question and verify AI-generated results. As AI continues to evolve, the study underscores the need for educational strategies that balance technology integration with the development of human mathematical and analytical skills.