The rapid integration of machine learning into applications in sensitive domains like healthcare and defense raises serious privacy and security concerns. These applications require strong privacy protections, as they rely on large amounts of sensitive data for both training and inference. Fully Homomorphic Encryption (FHE) offers a promising solution by allowing computations on encrypted data, thereby preserving confidentiality throughout the entire machine-learning pipeline. However, FHE only supports linear operations natively. This makes it hard to implement non-linear activation functions - an essential component of modern machine learning applications - under FHE constraints. In this study, we design, implement, and evaluate activation functions optimized for FHE-based machine learning applications. We focus on two widely used functions: the Square function and the Rectified Linear Unit (ReLU). Our experiments utilize the LeNet-5 and ResNet-20 architectures implemented with the CKKS scheme from the OpenFHE library. For ReLU, we compare two approaches. First, we explore the popular low-degree polynomial approximation approach. Second, we introduce a novel scheme-switching technique that also securely evaluates ReLU under FHE constraints. Our results show that the square function is highly effective in shallow models like LeNet-5, achieving 99.4% accuracy with an inference time of 128 s per image. In deeper networks like ResNet-20, ReLU is more appropriate. The FHE-based ResNet-20 model implemented with ReLU polynomial approximation resulted in an accuracy of 83.8% and an inference time of 1,145 s per image. Further, our proposed scheme-switching algorithm achieved a higher accuracy of 89.8%, with an increased inference time of 1,697 s per image on a ResNet-20 model as well. These results highlight the key trade-off to be considered when selecting activation functions for FHE-based machine learning applications. The activation functions that reduce computation time tend to significantly lower accuracy, while those that preserve accuracy incur higher computational costs.