Medical image classification is a critical component of modern healthcare, providing numerous advantages, including improved diagnostic accuracy and treatment planning. Integrating deep learning for medical image classification gives the ability to provide high accuracy and extract automated features. The residual connections, dilated convolutions, and attention mechanisms were introduced to enhance the performance of a very deep neural network. However, it remains challenging to achieve high classification accuracy in medical image classification tasks using neural network models with fewer trainable parameters and floating point operations per second (FLOPs). In this paper, we propose a lightweight neural network model for medical image classification, named as Medical Dilated Convolution and Attention Network (MeDiANet), which achieves better accuracy even with the fewer parameters and FLOPs as compared to the state-of-the-art (SOTA) models. The Dilated Residual Attention (DiET) is introduced in \(\texttt{MeDiANet}\) which provides the access to usage of different dilation rate based on the depth of the network. Also, by inflating the kernel size, more features can be extracted while keeping the parameter count low. The performance of MeDiANet has been evaluated on a large-scale multi-modal dataset. It achieves an accuracy of 94.18% with 0.38M trainable parameters, 0.08 FLOPSs, and lower inference time of 4.2 seconds which is on average, \(1.66\%\) higher in accuracy with \(12.50\times \) and \(2.55\%\) lower trainable parameter and FLOPs respectively, with \(7.44\%\) higher inference time compared to prior art.