With the rise of Machine Learning as a Service (MLaaS), the issue of privacy protection has also emerged. Homomorphic encryption can effectively protect the privacy of users and the service provider, but it requires high communication costs and computational complexity. To address these challenges, we propose a novel approximate matrix multiplication without multiplying model based on homomorphic encryption. This model can effectively protect user data and be extended to Multi-Layer Perceptron (MLP) networks without the need for bootstrapping operations. It improves the execution efficiency of matrix multiplication in machine learning and reduces the data communication overhead between users and the service provider. Our experiments show that single-image classification only takes 0.28 s, and for more complex networks, it only takes 1.01 s. Furthermore, through parameter pre-training lookup tables, we have achieved the effect of fitting multiple activation functions, with an accuracy of up to 96%. The experimental results demonstrate the potential of our model in privacy protection.