<p>Ultrasound (US) is widely used in clinical practice for the screening internal organs and guiding interventions. Nonetheless, US imaging suffers from inter- and intra-operator variations. Leveraging the reproducibility offered by robots, robotic ultrasound systems emerge as a promising solution, offering enhanced precision, stability, and repeatability. To realize autonomous US scanning, robot learning algorithms have been widely explored. However, current approaches primarily base the decision-making process for US navigation on 2D US images data, often overlooking the integration of 3D anatomical knowledge, which is a critical component for path planning in anatomically complex regions, such as the intercostal area. To address this limitation, we propose a novel reinforcement learning (RL) approach for intercostal US scanning path planning, leveraging computed tomography (CT) templates and utilizing 3D state representations. To this end, a virtual environment is developed using CT templates with randomly initialized tumors of various shapes and locations as a training environment. In addition, task-specific state representation and reward functions are introduced to encourage the convergence of the training process while minimizing the effects of acoustic attenuation and shadows during scanning. It is important to note that the scope of this work is limited to the autonomous path planning component, while robotic execution and control integration will be addressed in future studies. To validate the effectiveness of the proposed approach, experiments have been carried out on unseen patient models with randomly defined single or multiple scanning targets. The results demonstrate the efficiency of the proposed RL framework in planning non-shadowed US scanning trajectories in areas with limited acoustic access.</p>