This chapter introduce a shuffle and substitution-based defense mechanism (SSDM) to mitigate hardware-level threats and backdoor attacks. SSDM leverages pixel-level shuffling, neuron-level shuffling, and substitution techniques to enhance security without significantly degrading classification performance. Experimental evaluations demonstrate that SSDM effectively reduces the success rates of various hardware Trojan (HT) and backdoor attack types, providing a robust defense strategy for CNN accelerators while maintaining an acceptable trade-off with model accuracy.