Knowledge graph (KG) embedding aims to learn vector representations for entities and relations. To support distributed and secure training, federated KG embedding collaboratively learns an embedding model among multiple organizations without directly sharing raw data. A federated KG embedding framework should be capable of incorporating both learning and unlearning paradigms, due to the dynamic nature of KGs and the potential need for visibility adjustment by KG owners. However, there are still several challenges in developing a federated KG embedding framework, including overcoming the communication and computation overhead caused by the huge number of entities and controlling the scope of influence when unlearning. In this paper, we propose a novel parameter-efficient federated KG embedding learning and unlearning framework, named PFLU. Specifically, we incorporate an anchor-based federated KG embedding technique to reduce computation and communication overhead when transferring knowledge. We address the anchor selection problem by formulating it as a maximum coverage problem and designing a greedy strategy for its resolution. Besides, to achieve a desirable balance between propagation and retention of unlearning effects, we adopt a two-stage mechanism combining structural and semantic unlearning. Extensive experiments on widely-used datasets show the superior parameter efficiency of PFLU over several baselines in both federated KG embedding learning and unlearning. Furthermore, we provide empirical evidence and discussions to show the effectiveness of the proposed anchor selection strategy.