Scholarlink

<p>The rising prevalence of Large-Language Model-based AI Services (LLMAIs) underscores the importance of understanding users’ self-efficacy and specific needs in engaging with these technologies. Such insights are pivotal for evaluating LLMAI instructional effectiveness and guiding future developments of these technologies. However, current instruments fail to adequately capture these aspects. The study aimed to develop and validate the Questionnaire of Self-efficacy and Needs in using core features of LLMAIs (Q-SNELL). The Q-SNELL was developed through an extensive review of LLMAI core features. Its validation involved content and face validity assessments with a diverse group of experts and common users. Convergent validity, known-groups validity, and test-retest reliability were further evaluated through first and second round online surveys completed by users of LLMAIs. The Q-SNELL comprises three parts: the first focuses on self-efficacy and needs related to the eight core features of LLMAIs; the second includes two items identifying the most frequently used features; and the third assesses general self-efficacy. Content validity showed high agreement (76–100%), while face validity was also strong (80–100%), except for the ease of responding to the scale design (59% and 68% agreement). Validation with 398 participants showed mixed outcomes for convergent validity, strong support for known-groups validity, and moderate to high test-retest reliability (ICC = 0.47 to 0.74). The Q-SNELL is a pioneering instrument measuring users’ self-efficacy and needs regarding LLMAIs’ 8 core features and general usage. Its flexible structure allows for selective application of specific items, tailored to the needs of researchers.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Developing the questionnaire of self-efficacy and needs in using large-language model-based AI services

  • 期刊论文

  • 发表日期：

    2025年1月4日

查看全文
  • Yu-Jeng Ju,
  • Yi-Ching Wang,
  • Shih-Chieh Lee,
  • Cheng-Heng Liu,
  • Jen-Hsuan Liu,
  • Chih-Wei Yang,
  • Ching-Lin Hsieh

摘要

The rising prevalence of Large-Language Model-based AI Services (LLMAIs) underscores the importance of understanding users’ self-efficacy and specific needs in engaging with these technologies. Such insights are pivotal for evaluating LLMAI instructional effectiveness and guiding future developments of these technologies. However, current instruments fail to adequately capture these aspects. The study aimed to develop and validate the Questionnaire of Self-efficacy and Needs in using core features of LLMAIs (Q-SNELL). The Q-SNELL was developed through an extensive review of LLMAI core features. Its validation involved content and face validity assessments with a diverse group of experts and common users. Convergent validity, known-groups validity, and test-retest reliability were further evaluated through first and second round online surveys completed by users of LLMAIs. The Q-SNELL comprises three parts: the first focuses on self-efficacy and needs related to the eight core features of LLMAIs; the second includes two items identifying the most frequently used features; and the third assesses general self-efficacy. Content validity showed high agreement (76–100%), while face validity was also strong (80–100%), except for the ease of responding to the scale design (59% and 68% agreement). Validation with 398 participants showed mixed outcomes for convergent validity, strong support for known-groups validity, and moderate to high test-retest reliability (ICC = 0.47 to 0.74). The Q-SNELL is a pioneering instrument measuring users’ self-efficacy and needs regarding LLMAIs’ 8 core features and general usage. Its flexible structure allows for selective application of specific items, tailored to the needs of researchers.