With an increasing volume of data being collected and processed on edge, Federated Learning (FL) has become a dominant approach to train shared global models on distributed data held by diverse edge clients without exposing privacy. However, with expectations of high-quality user experience, instantaneous intelligent decisions must be made at the network edge to meet performance, security, and latency requirements. Intuitively, this challenge becomes more daunting as edge devices are constrained by limited computing and storage resources. Furthermore, edge clients need to continuously acquire, update, accumulate, and apply knowledge in domains requiring the processing of sensitive data and long-term model training. It has been proven that edge clients often forget previous tasks knowledge after learning new tasks, resulting in catastrophic forgetting. To mitigate this, we propose a Federated Meta Continual Learning (FMCL) approach. It requires edge clients not only to learn new tasks quickly but also to memorize rapidly. Specifically, we adopted a MAML (Model-Agnostic Meta-Learning) framework with memory replay, setting up buffers at edge clients to consolidate old tasks. In such a way, the model learns optimal initialization to quickly adapt to all seen tasks while preventing catastrophic forgetting. Extensive evaluation results indicate that FMCL performs well in alleviating catastrophic forgetting and achieves a good balance between learning new tasks and remembering existing tasks.