Existing multimodal Point of Interest (POI) classification methods mainly used convolutional neural networks to extract visual features and employed simple strategies for feature fusion, neglecting the impact of contextual semantics. This paper proposes a POI classification system based on the large multimodal model CLIP, named GA-CLIP, which includes two stages: knowledge-enhanced feature learning and POI classification decision. Among them, in the multimodal feature learning stage, the system employs a gated attention mechanism to integrate textual, visual and emotional features for fine-grained cross-modal interaction, while incorporating a prompt engineering component that leverages a local knowledge base with dictionary definitions as prior knowledge to enrich sparse textual representations and enhance classification robustness. In the classification decision stage, GA-CLIP leverages the large multimodal model to analyze emotional features to enhance the interpretability of classification decisions. Experimental results validate the GA-CLIP system’s robust classification performance (F1 score: 65.82) and its applicability in POI classification tasks.