<p>In various applications, such as access to mobile device, web application access, etc., user’s voice biometric authentication is one of the easy and excellent authentication factors for these applications. It can increase user’s usability, be believed to provide enhanced security unlike PINs, and improves customer experience. However, generally, in the authentication process, false acceptance is one of the fatal weaknesses since it leads to system access for the unauthorized person. Especially, in the case of the mobile environment with only a small training dataset, it is very hard to reduce the fault acceptance rate. To address this limitation of user’s voice biometric authentication, in this paper, we propose a novel approach that dramatically reduces the weakness of mis-acceptance from a given deep learning-based voice authentication with a small training dataset. To prove this improvement, we experimentally show that all test samples that were mis-accepted in a given deep learning-based voice authentication trained with a small dataset are correctly validated after applying our technique.</p>