Early diagnosis of Alzheimer’s disease (AD) is critical for timely intervention and management. Handwriting analysis has been recognized as a promising diagnostic tool, as it is one of the first skills affected by AD. This research investigates the potential of using deep learning to assist AD diagnosis through handwriting image analysis. Unlike previous methods applied on this dataset, the model is trained on images of different handwriting tasks instead of training task-specific models. This approach improved the model’s ability to identify AD-specific writing patterns and enhanced diagnostic performance. We conducted an extensive analysis of various convolutional neural networks and vision transformers, focusing on the ability of transformer models to transfer knowledge across different domains. Our contributions are as follows: (i) we conducted comprehensive experiments to identify the most effective deep learning models for the AD handwriting task, providing a benchmark for future research; (ii) we redesigned a pre-trained Optical Character Recognition (OCR) transformer model to assess its capability to transfer knowledge between OCR and classification tasks; and (iii) we developed a three-stage framework to increase performance and evaluate the impact of distinct handwriting tasks on the final AD diagnostic outcome. Our framework surpasses previous methodologies, achieving an accuracy of 87.99% and a sensitivity of 89.69%, demonstrating that handwriting-based deep learning-aided diagnostic systems hold significant promise as tools for early AD detection.