Leveraging the powerful text generation capabilities of large language models (LLMs) has the potential to enhance the performance of classification models through data augmentation. However, few studies have demonstrated notable improvements in accuracy for multi-class classification tasks, such as news article classification, using this approach. In this study, we investigate whether augmenting training data with LLM-generated documents can improve classification accuracy in the context of news categorization. Specifically, we explore a method in which category-specific keywords are provided to an LLM to generate new documents, which are then added to the training set. Contrary to expectations, this approach resulted in a decline in classification accuracy. Further analysis of the embedding representations of the augmented data revealed that the generated documents had a distribution that differed from the original training data. This mismatch introduced noise during classification, likely contributing to the observed performance degradation. These findings suggest that while LLM-based data augmentation holds promise, careful attention must be paid to the alignment between generated and original data distributions—particularly in multi-class classification tasks involving real-world textual data such as news articles.