Speech synthesis is striving to explore efficient content representations for different tasks, which also drives research on models capable of performing multiple tasks. However, current approaches are predominantly based on text-to-speech models, which require large-scale transcribed datasets, simultaneously limiting the representations. To address this issue, we propose EmbSpeech, a novel framework unifying zero-shot speech synthesis tasks through speaker-independent self-supervised speech representations. EmbSpeech is built on a voice conversion model and consists of three task-specific encoders, a speaker encoder, and a diffusion-based decoder. With task-specific encoders, EmbSpeech can perform voice conversion, text-to-speech synthesis, and speaking style conversion. Experiments show that our framework can achieve naturalness and intelligibility comparable to or even better than the baselines in low-resource settings.