One of the main obstacles in speech pathology research is the limited availability of data resources. This limitation has restrained the use of deep learning models in this domain even though they have been achieving SOTA results, since they heavily rely on large datasets. This limitation impacts the development of reliable tools for assessing speech disorders such as dysarthria. To address this challenge, we explore the use of pre-trained models that can be fine-tuned with limited data. Specifically, we fine-tune two spectrogram-based pre-trained models, Whisper and AST, for dysarthria severity classification. An ablation study is conducted to analyze model performance and generalization using the UASpeech and TORGO datasets under a speaker-independent setup, with both in-dataset and cross-dataset testing. Whisper achieved 46% in-dataset and 67% cross-dataset accuracy, while AST achieved 45% and 59%, respectively. These results indicate that pre-trained models, particularly Whisper, can effectively generalize across datasets and offer a promising direction for improving dysarthria severity classification in low-resource settings.