Multimodal Analysis of Human-AI Collaborative Design: Eye-Tracking and Think-Aloud Insights into Cognitive Transitions
摘要
This study examines the cognitive dynamics of collaborative design by comparing human-AI collaborative environments with traditional sketching through multimodal analysis. By integrating eye-tracking data and think-aloud protocol coding, we investigated transitions between cognitive states (Problem Solving, Evaluation, Operation, and Naivety) during the creative process. Four design experts with architectural backgrounds participated in a mixed within/between-subject experiment. Results reveal distinct patterns: human-AI collaborative design promotes prolonged fixation and evaluation-driven cognitive cycles, fostering reflective interpretation of AI outputs; traditional design encourages exploratory visual behavior and direct shifts from ideation to execution. Transition matrices quantify these differences, demonstrating how collaborative AI systems reshape designers’ cognitive rhythms. These findings provide empirical foundations for developing more cognitively aligned human-AI collaborative systems and offer a novel framework for analyzing human-AI interaction in creative contexts.