Robotic manipulation has always been a great challenge in the area of robotics. Currently, most methods still rely on some computer vision technologies to detect grasping points regarding the type of robotic hand. However, the real-world environment is usually unstructured, and it happens that the target object is occluded or its pose is unsuitable for grasping. In this situation, the robotic hand is expected to have the ability to actively explore the environment. In this chapter, we propose a new task of active exploration for robotic manipulation. A composite robotic hand composed of a gripper and a suction cup is used to explore a complicated environment in order to grasp the target object. The exploration strategy is obtained by the proposed deep reinforcement learning framework. Case studies are conducted on real robotic platform illustrating the effectiveness of the proposed active exploration for robotic manipulation.