Given a target variable of a dataset, existing Federated Local Causal Structure Learning (FedLCS) approaches aim to learn direct causes and direct effects of a given target variable from decentralized data across multiple clients while considering data privacy. Since a current FedLCS approach implements iteratively conditional independence (CI) tests for skeleton construction and edge orientation from decentralized data, its computational efficiency remains a limitation. To address this limitation, in this paper, we propose an efficient FedLCS algorithm called eFedLCS that integrates GPU-accelerated computation into CI tests to speed up federated skeleton learning and skeleton orientation. Specifically, we design GPU-based \(G^2\) tests and Fisher-Z tests to reduce computational costs of CI tests. In addition, we design a caching mechanism on each client to store previously executed CI tests for minimizing redundant computations for further improving the efficiency of eFedLCS. Based on the accelerated tests, we propose a novel strategy to efficiently identify correct separation sets for skeleton orientation. Experiments with benchmark datasets have shown that eFedLCS is at least 3.6 times faster than FedLCS without significant degradation in structure learning performance.