Neural network cloud service is now a wide-ranging application for inference to reduce local computing pressure. However, if the data and model are sensitive, the service can cause privacy and security issues. Homomorphic encryption is an important means to protect privacy because it can be computed directly on the ciphertext. In this paper, we design an efficient privacy-preserving neural network cloud service (PNNCS) system. First, based on the federated learning framework, we adopt CKKS homomorphic encryption and introduce a third-party aggregation center to protect the security of parameter transfer in the training phase. Second, the privacy prediction between the user and the cloud service provider is carried out between the weighted-plaintext and the input-ciphertext, and then we introduce a packed encoding method to reduce homomorphic operations. Experimental results show that the proposed scheme can effectively reduce the communication cost and improve the computational efficiency in the context of privacy security.