Machine Learning as a Service (MLaaS) become a popular application in cloud-based services, where users upload their data to a service provider and the provider runs a pre-learned model to give predictions back. However, MLaaS has a risk of privacy leakage cause the data transferred may contain users’ sensitive information. Homomorphic Encryption (HE) is a promising way to handle this problem. It allows the server to make calculations directly on encrypted data, and so guarantees the confidentiality of data. Unluckily, inference on encrypted data usually has massive time consumption and high memory requirement. To solve this problem, we propose Mini-batching to increase the batch utilization of ciphertext, which allows adjustment to models’ performance levels. Each level has its time cost, memory requirement, and classification capacity, indicating the number of images classified per inference. Additionally, we design a protocol called Simulating ReLU via Homomorphic Encryption (SRHE) to implement the real ReLU function on ciphertext which proved to be suitable for deeper neural networks. In the experiment, we implement our model Noya on MNIST and CIFAR-10 datasets for six performance levels and achieves 99(76) % accuracy. Among them, Noya-1 infer 32 images in 7.7(267) s, whose amortized time is 0.24(8.34) s, outperforming most other secure models with HE. The fastest version of Noya, Noya-32, achieves 0.62(13.24) s which speeds up 2(8) times compared with prior work.