Deep Convolutional Neural Networks (CNNs) and Vision Transformers excel in computer vision tasks like leaf disease detection, brain tumor detection, and X-ray classification. These models rely on datasets to learn and classify features accurately. However, acquiring large datasets for training and validation is challenging, particularly in medical image analysis. Even large quantities of image data with limited system resources can hinder performance by preventing the fitting of the dataset into CNN models. Image size significantly affects training time, often necessitating cropping high-resolution images to manageable sizes to avoid memory issues. This survey paper discusses various neural network models for different dataset sizes, highlighting methods for data augmentation and advancements in neural network design. For small datasets, models like prototypical networks with a pre-trained ResNet18, episodic learning with ResNet-12, and vision transformers show superior performance, achieving nearly 99% accuracy. For moderate to large datasets, DenseNet, vision transformers, and ResNet50 consistently achieve 93% accuracy. The github link ( https://github.com/mrinmoysadhukhan/Surveyppaper ) is provided containing link of the disucssed dataset.