In this paper, we present a lightweight symmetric encryption scheme specifically designed to support homomorphic operations with respect to constant multiplication. The scheme is based on the principles of singularization, a moving target defense strategy designed to safeguard resource-constrained devices. It allows clients to outsource linear computations to remote servers while requiring only a simple addition operation for encryption, making it highly efficient and well-suited for devices with limited resources. Additionally, we propose a remote key generation protocol to address scenarios in which the client lacks the computational capacity to generate keys. Building on this encryption scheme, we develop a protocol for privacy-preserving neural network inference, particularly for cases where the model parameters are public but the network is accessed through a service. Experimental results demonstrate that our protocol achieves greater efficiency compared to existing approaches based on secure multi-party computation or homomorphic encryption.