In the evolving landscape of Natural Language Processing (NLP), the rapid advancement and success of Large Language Models (LLMs) have been a major talking point, particularly with their potential to outperform existing state-of-the-art technologies in various NLP tasks. Among these tasks, anonymization of unstructured, textual data represents a critical challenge, traditionally addressed by Named Entity Recognition (NER) models. This paper investigates the hype surrounding LLMs, specifically evaluating their effectiveness and applicability in the context of support ticket anonymization by conducting a comparative evaluation, pitting LLMs against an established support ticket anonymization solution that utilizes state-of-the-art transformer architectures for textual data anonymization. Our findings reveal significant limitations of LLMs in terms of overall performance and accuracy, particularly when facing real-world anonymization scenarios.