Autonomous elevator operation promises enhanced robot inter-floor navigation. However, existing methods either require adapting elevators for wireless communication or rely on complex and computationally expensive approaches (involving repetitive button detection and character recognition). This paper proposes a method that leverages a robot’s initial learning of the elevator panel layout to identify button functions based on their relative positions, avoiding redundant recognition tasks. First, the robot learns its layout (button location and function) by building a graph representation from a given template. Second, a two-stage template matching process efficiently locates the panel, followed by target button identification based on the pre-learned structure. An autonomous distortion removal step refines recognition for high accuracy, especially under varying viewpoints. Experiments validate the effectiveness of this approach, demonstrating significant improvement in recognition rates, particularly with severe perspective distortion. Binocular vision enables 3D target button localization, which is required to complete the autonomous elevator operation task.