End-to-end robotic manipulation learning techniques have become one of the most prominent research areas in recent years. Imitation learning allows the robot to learn from expert demonstrations, which helps reduce the costs of exploration and trial-and-error. This paper focuses on addressing the robot manipulation tasks using imitation learning. An improvement based on the diffusion policy with point clouds as the visual representation is developed. More specifically, we redesign the point-cloud sampling strategy to give higher priority to points that belong to the manipulated object and its local geometric details. In addition, an attention-driven point cloud encoder to capture long-range spatial dependencies is proposed. To introduce adaptive, channel-wise attention, we embed Squeeze-and-Excitation (SE) modules at critical stages of the U-Net backbone. Simulation examples, several manipulation tasks, are provided, and the proposed method attains higher success rates on a greater proportion of tasks across the selected test suite.