Speech classification models are extensively utilized and significant in various domains. However, recent research has demonstrated their susceptibility to backdoor attacks, which can lead to security risks. Many traditional methods based on data poisoning are prone to detection for they involve manipulating data and labels during both the training and inference phases. In this paper, we introduce semantic backdoor attacks based on code poisoning by training the main task and backdoor task. We propose a phoneme mixture and multiple-task learning strategy to implement blind backdoor attacks on classification tasks. In this scenario, the attacker does not need to alter the training data and ensures the model predicts wrongly in the inference stage without poisoning the input sample, showing great stealthiness. The phoneme mixture uses the attacker-specific phoneme as semantic triggers and mixes it with training speech samples, leveraging the inherent phonemes or syllables present in the speech samples to activate the backdoor without input modification during the inference phase. Also, the poisoning code will dynamically pollute the training inputs. In this case, the model needs to optimize both the main task and the backdoor task at the same time, so we use the Multiple Gradient Decent Algorithm (MGDA) to optimize the losses generated by these two tasks at the same time so that both tasks can achieve higher accuracy. Our experiment shows that the accuracy of the attack success (ASR) is close to that of poisoning-based backdoor attacks on speech classification.