In medicine and clinical research, a biomarker serve as a measurable indicator for assessing health status and monitoring disease progression. Using retinal images to predict biomarkers for non-retinal diseases such as cardiovascular disease and chronic kidney disease has shown promise. However, despite the success of utilizing retinal images, significant challenges remain. One major issue is the limited availability of retinal images with linked health screening records, as collecting such coordinated data is logistically complex and resource intensive. To address data scarcity, we investigate whether more generalized features than image-level features can be learned by matching retinal images from the same patient. We propose a patient-level contrastive learning approach that defines positive pairs not only at the image level but also extends to the patient level. This method proves effective in predicting biomarkers, outperforming existing image-level methods in biomarker detection and survival analysis tasks, thereby enhancing overall model consistency across various types of retinal imaging.