Vector fonts are favored by font designers for their editability. However, reconstructing vector images from raster glyph images is a cross-modal process. Existing methods either struggle to reconstruct fine contours or fail to represent glyphs concisely. This paper proposes a dual-modal vector denoising diffusion model that integrates vector and raster images for reconstructing quadratic Bézier curves from raster images. Specifically, we pre-calculate the signed distance function (SDF) values for each input glyph to effectively capture the geometric information of the glyph. Then, we integrate the image and vector dual modalities to enhance the stability of the reconstruction process. By incorporating a variable masking mechanism, the model gradually reduces its dependency on the vector modality, enabling end-to-end cross-modal vector reconstruction during the inference process. Our vector diffusion model employs a transformer architecture and an innovative vector representation method, capable of modeling various vector geometric shapes. In the realm of vector font reconstruction, our approach outperforms existing techniques. The font images generated by our method can be readily converted into TrueType fonts, highlighting the significant practical value of this research.