Background <p>As artificial intelligence (AI) is increasingly introduced into endoscopic navigation, understanding how operators use and respond to AI suggestions is essential for designing safe and effective human-AI collaboration. This study investigated the effects of AI reliability and operator experience on task performance and cognitive responses in a simulated colonoscopy navigation task.</p> Methods <p>A virtual simulation paradigm was used, in which 27 interns and experienced doctors completed a navigation direction judgment task under three AI reliability conditions (high, medium, and low) and a no AI condition. Navigation accuracy, task completion time, subjective trust, cognitive load, and compliance with incorrect AI prompts were recorded.</p> Results <p>The results showed that AI reliability significantly affected navigation performance and subjective trust. High reliability AI improved task accuracy and efficiency, particularly among less experienced operators. However, when AI reliability decreased, participants’ performance declined, and interns showed higher compliance with incorrect AI prompts than experienced doctors. Notably, this behavior occurred even when subjective trust decreased, suggesting a potential mismatch between trust evaluation and AI following behavior. This could be interpreted cautiously, as higher compliance among interns may reflect both rational reliance under high task difficulty and the risk of inappropriate reliance when AI reliability is insufficient.</p> Conclusions <p>These findings indicate that AI-assisted endoscopic systems should not only improve navigation performance but also support appropriate reliance calibration. Differentiated support strategies may be needed for operators with different levels of experience, especially in scenarios involving training or novice operators.</p>