Imaging genetics, discovering associations between imaging and genetic variations, has emerged as a promising avenue to advance the understanding of neurological disorders. However, the majority of existing studies focus on selecting disease-related features to improve prediction accuracy using statistical analysis or learning-based methods. Despite the data-intensive nature in medical imaging, understanding of how genetics affect brain structures in image generation remains largely unexplored. In this paper, we propose a novel approach that generates brain images from genetics leveraging latent diffusion models. Specifically, attention-based diffusion models are conditioned on genetic information, which allows us to enhance the quality and relevance of the generated images in the context of Alzheimer’s diagnosis (AD). We validated our model on T1 MRI and single nucleotide polymorphism (SNP) in the Alzheimer’s Disease Neuroimaging Initiative (ADNI). Our model yields real-like synthetic images demonstrating AD-specific variation that helps to increase accuracy in a downstream classification of AD. Overall, our study highlights the potential of diffusion models in imaging genetics to facilitate accurate diagnosis and understanding of AD.