Machine Learning as a Service (MLaaS) has been widely adopted, offering convenient AI services for resource-constrained environments. To address privacy concerns in MLaaS, there has been increasing attention on privacy-preserving inference for Convolutional Neural Networks using homomorphic encryption. However, existing methods often overlook model parameter quantization and non-arithmetic functions, which can degrade accuracy as the model depth increases. This paper proposes a privacy-preserving neural network inference scheme using binary neural networks and the TFHE scheme. We introduce optimizations based on bootstrapping reduction and operational logic, which significantly reduce computational overhead while maintaining accuracy. Experimental results show that the proposed optimization strategy reduces network runtime of MNIST from 920.3s to 445.3s, representing a 51.7% reduction in overall computational time, while achieving 94.63% accuracy on CIFAR-10.