Deep neural networks are susceptible to backdoor attacks, where the attacker just need to poison a small fraction of training data such that the attacked model work normally on benign samples but output the target label on poisoned samples. Most existing attacks inject their trigger in pixel space, which tend to be noticeable to human eyes. Some attacks inject their trigger in frequency domain and their malicious samples are stealthy, but they are not resistant to current backdoor defenses. In this paper, we propose a simple but effective attack method, dubbed as Hadamard transform based backdoor attack (HTBBA). In our attack method, we only need to poison a small portion of training samples, while no need to modify other training components like model structure or training loss. Specifically, inspired by Hadamard transform based image watermarking, we generate invisible backdoor trigger dispersed across the entire image in frequency domain by embedding an attacker-specified image into benign samples through Hadamard transform. Extensive experiments are conducted on benchmark datasets under different settings, demonstrating the effectiveness and stealthiness of our method and its resistance to various backdoor defenses.