A Standardized Multimodal Dataset Recording Protocol for Early Cognitive Impairment Screening
摘要
We present a standardized protocol and dataset design for multimodal acquisition in older adults with suspected cognitive impairment, aimed at supporting early identification research and data-intensive machine learning. The protocol combines four time-bounded discourse elicitation tasks, personal narrative, picture description, story narration, and procedural discourse, to capture complementary linguistic and cognitive demands. Speech is recorded with an external microphone and transcribed to provide aligned audio–text modalities, while an RGB camera facing the participant records facial behavior to enable analysis of nonverbal cues. In addition, we integrate an immersive VR/MR serious-game module inspired by our HoloDemTect framework, including daily-life activities such as a shopping-list task that requires selecting target items and placing them into a box. The VR/MR environment is fully instrumented to retain fine-grained telemetry (e.g., completion times, step-level latencies, interaction events, and error patterns), and gaze-related measures are recorded when supported by the hardware. To provide clinically meaningful reference outcomes for operational stratification and downstream classification, each session includes a brief self-assessment of cognition and instrumental functioning using Test Your Memory and the Lawton & Brody IADL scale. Overall, the protocol yields a structurally complete and diverse dataset that enables joint analyses of speech, language, facial behavior, and immersive task performance, addressing common unimodality constraints in existing resources and facilitating multimodal deep learning baselines and future benchmarking.