Generative artificial intelligence (gen AI) is becoming increasingly popular. However, evaluating it with a multi-faceted, but easy-to-apply model remains a challenge. By cross-referencing findings from information systems (IS) success research, we elaborate on such a model. Taking a case study within a science & technology company, we reveal five takeaways: (1) Gen AI often starts with conversational agents. Its benefits are best measurable by net present value; (2) Enhance “pure” efficiency evaluations with effectiveness criteria; (3) With an experience category take user’s learning curve into account; (4) Towards green IS, think about sustainability as a model’s fourth value category; (5) In a summary, multi-faceted evaluation models should take efficiency, effectiveness, experience, and sustainability criteria into account.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluating Gen AI Use Cases: Takeaways from a Case Study

  • Jörg H. Mayer,
  • Kim Erik Albrecht,
  • Reiner Quick

摘要

Generative artificial intelligence (gen AI) is becoming increasingly popular. However, evaluating it with a multi-faceted, but easy-to-apply model remains a challenge. By cross-referencing findings from information systems (IS) success research, we elaborate on such a model. Taking a case study within a science & technology company, we reveal five takeaways: (1) Gen AI often starts with conversational agents. Its benefits are best measurable by net present value; (2) Enhance “pure” efficiency evaluations with effectiveness criteria; (3) With an experience category take user’s learning curve into account; (4) Towards green IS, think about sustainability as a model’s fourth value category; (5) In a summary, multi-faceted evaluation models should take efficiency, effectiveness, experience, and sustainability criteria into account.