In this research, we introduce a fine-tuned multi-head self-attention framework designed to address the cocktail party effect, also known as speech separation. The integration of intra- and inter-segment, along with multi-head self-attention, represents a notable advancement in effectively separating lengthy time-domain speech signals, serving as a baseline architecture. We implement transfer learning methods for each layer of the model with scheduling mechanisms that adjust the learning rate across different datasets. This approach allows the model to be refined and updated using prior knowledge, effectively boosting performance while reducing training time and cost. It is especially useful for adapting existing models to similar tasks or enhancing their effectiveness. The fine-tuned model outperforms non-fine-tuned ones by reusing learned features from earlier training stages. Experimental results indicate that the proposed method achieves better performance than the baseline framework and surpasses existing approaches.