In the current field of natural language processing, sentiment analysis faces the challenge of multimodal data fusion, especially in combining text sentiment with speech sentiment to improve analysis accuracy. This paper aims to realize multi-source data fusion of text and speech by combining large models with deep learning technology to improve the accuracy and application breadth of sentiment analysis. This paper uses a pre-trained language model (BERT) to extract sentiment features from text data, processes speech signals through a convolutional neural network (CNN), extracts acoustic features, and finally jointly models text and speech features through a deep fusion network to form a comprehensive sentiment representation. In the test, the accuracy of sentiment analysis based on the fusion model reaches 96.6%. This study not only provides a new multimodal fusion method for sentiment analysis but also provides a theoretical basis and practical guidance for related applications such as intelligent customer service and social media monitoring.