The hand plays a pivotal role in human grasping and objects manipulation, which is the key for successfully performing downstream tasks. Existing methods rely on human intention or task-level text descriptions but lack explicit control over which object part to grasp. To address this challenge, we propose Text2Grasp, which utilizes both template and personalized text prompts to guide grasping specific object parts, enabling precise control over the grasp synthesis. Specifically, we initially apply a text-guided diffusion model to generate a coarse grasp pose, then perform hand-object contact refinement that incorporates both finger and object part perception to further enhance grasp stability, accuracy and diversity. Extensive experiments demonstrate that our method can accurately control and synthesize reasonable hand grasps on specific object parts as instructed by text prompts.