Recognizing human daily actions under low-resource conditions remains a significant challenge due to limited labeled data and the inherent complexity of human motion. In this paper, we propose a novel transfer learning framework that enhances recognition performance in such scenarios by incorporating topological knowledge memory. Specifically, we construct a graph-based topological representation of the feature space in a well-annotated source domain using unsupervised learning. This representation captures the intrinsic structure of action features and serves as a reusable knowledge memory for transfer learning. In the target domain, the learned topological memory is combined with extracted features to guide the prediction process, enabling implicit knowledge transfer without requiring explicit feature alignment. To further improve adaptability and robustness, we introduce a gated attention fusion module that dynamically integrates multiple semantic representations. Our method effectively exploits transferable structural patterns from the source domain to support recognition tasks in low-resource target domains. We validate our approach by transferring knowledge from a public dataset to a self-collected dataset. Experimental results show that our method significantly outperforms conventional fine-tuning and feature extraction strategies, achieving higher accuracy and superior performance across multiple evaluation metrics. These findings highlight the potential of topological memory in advancing transfer learning for human action recognition in low-resource settings.