Source code summarization automates the generation of natural comments, enhancing efficiency in software development and maintenance. With the emergence of large language models (LLMs), significant progress has been made in this domain. However, current approaches either rely on manually crafted prompts or standard fine-tuning that fail to fully leverage the power of continuous embeddings. To address this gap, we propose StageCS, a novel framework integrating prompt learning and model fine-tuning for code summarization, featuring a strategic two-stage training process with a multi-branch transformer architecture. StageCS generates specialized continuous embeddings that synergistically guide LLMs to produce high-quality summaries while maximizing the benefits of targeted fine-tuning. Evaluations on the CodeSearchNet Java dataset show StageCS outperforms baselines in representative LLM architectures like PolyCoder and CodeGen. More importantly, our ablation studies demonstrate that both the multi-branch transformer and the strategic two-stage process contribute significantly. StageCS enables LLMs to excel in code summarization without extensive manual prompt engineering, delivering superior quality through specialized training.