Deepfake audio is synthetic voice generated using sophisticated artificial intelligence. It mimics human speech patterns, intonation, and voice characteristics. The deepfake audio undermines trust, authenticity of information, and security by spreading misinformation and engaging in social engineering attacks. To mitigate the risk, we proposed an optimized and generalized deep learning architecture to classify authentic human audio and AI-generated cloning voices. We used LSTM and its variant, BiLSTM, with optimal optimization and regularization on the benchmark dataset. The authentic recorded audio is used as authentic and then transformed using AI. We approach it as a binary classification problem and use different statistical analyses to distinguish variation in the distribution of temporal characteristics. The proposed architecture successfully captures audio data’s local spectrum information and long-range temporal relationships. We used different performance evaluation matrices to check the performance, and overall, we got around 99% accuracy. This framework can handle a variety of audio sources and is resistant to advanced deep-fake techniques.