<p>This paper provides a large-scale empirical study of application of supervised classifiers on extremely small-data binary problems. We benchmark some of the most traditional classifiers together with not commonly used classifiers based on (i) a Neural Tangent Kernel (NTK), (ii) a Support Vector Machine (SVM) with a kernel defined by an NTK, (iii) a Neural Network Gaussian Process (NNGP), and (iv) a Variational Auto-Encoder (VAE) because they already have shown some promising results in small data regime. In particular, we examine whether the classification defined as a posterior probability density function, calculated by a denoising VAE (DVAE) using the variational principle, can achieve the state-of-the-art in our scenario. We show that this is the case and that the DVAE outperforms all compared classifiers on low-dimensional data. We also observe that the DVAE classifier has superior precision in settings that require high sensitivity (recall) and that NNGP and NTK tend to become comparatively stronger as the training size decreases.</p>