the development of deep learning models is a highly lucrative proposition for many medical and diagnosis applications due to the abundance of imaging data. In certain cases however, the real-world implications of the diagnosis task can pose serious challenges to the training of such models despite their high promise for performance. In particular, deep learning models become increasingly difficult to train for the case of rare diseases such as Pulmonary Neuroendocrine Tumor (PNET) where the number of true negative samples are considerably larger than true positives. To address these challenges, in this work a new framework is proposed using Multi-Instance Learning (MIL) for the prediction of recurrence outcomes in rare disease diagnosis. In this approach, leveraging the attention weights of a Vision Transformer as a mechanism for upsampling under-represented patches, proves to mitigate the effect of dataset imbalance at patch level. Through experimentation, it is shown that applied to Tissue Micro-Arrays (TMA) obtained from hematoxylin and eosin (H&E) biopsies of PNET patients, the proposed framework can achieve precision and recall values as high as 0.90 and 0.92.