Legal judgment prediction (LJP) is a crucial task in intelligent judiciary systems. We observe that existing LLMs perform suboptimally in this task. The main challenge lies in the inherent conflict between the abstract labels and the lengthy textual facts, making it difficult for LLMs to reason accurately. To enable LLMs to adapt effectively to the unfamiliar LJP task, we propose a novel framework for Chinese LJP, termed N2RPT, which draws inspiration from the reasoning processes of real-world judges and leverages a sophisticated integration of legal norms to enhance decision-making precision. N2RPT employs a pre-trained language model (PLM) collaborates with a LLM through an iterative, relevance-driven retrieval process that refines information from coarse to fine granularity. Subsequently, strict label-consistent legal norms are employed as candidates and demonstrations within prompt engineering, ensuring that the LLM adheres to established legal standards during the reasoning process. To further mitigate the risk of hallucinations in LLM outputs, GPT-4 is leveraged to synthesize reasoning trajectories, which are then used to fine-tune the LLM and enhance its capability. Extensive experiments conducted on real-world datasets demonstrate the effectiveness and superior performance of the proposed framework in enabling LLMs for LJP task.