Effectively modeling topics in short texts, such as tweets and news snippets, is crucial to understanding rapidly evolving social trends. However, existing topic models struggle to capture the underlying semantic patterns of short texts due to their inherent data sparsity. This characteristic leads to insufficient co-occurrence information, impairing the coherence and granularity of extracted topics. To address these challenges, we introduce Topic Refinement, a novel model-agnostic mechanism for short-text topic modeling that leverages the advanced text comprehension capabilities of Large Language Models (LLMs). Unlike traditional approaches, this post-processing mechanism employs prompt engineering to refine topics across various modeling methods. We guide LLMs to identify semantically intruder words within the extracted topics and suggest coherent alternatives to replace them. This process emulates human-like identification, evaluation, and refinement of the discovered topics. Extensive experiments on four diverse datasets demonstrate that Topic Refinement improves the topic quality and performance in topic-related text classification tasks.