Optic Atrophy is a significant cause of visual impairment and demands accurate and timely diagnosis for effective treatment. However, diagnosis of this condition is often carried out by manual inspection by ophthalmologists. Recent studies have attempted to automate this diagnosis using machine learning techniques from fundus images. The prime challenge in this approach lies in the limited availability of public datasets and the difficulty in handcrafting distinctive patterns in the fundus images. The present work proposes using a Siamese network architecture for Optic Atrophy detection using fundus images, which would be effective despite the limited number of publicly available samples of the condition. The other contribution of the paper is the use of augmentation techniques, viz., random Gaussian blur, and the addition of salt-and-pepper noise to increase the generalization of the problem. The Siamese network accepts pairs of inputs and classifies the images as similar or dissimilar according to the distance of the two instances in the embedding space. One of the images is taken as a standard image, while the second is the test image. The proposed method provides better results than the existing works.