Performance and Self-evaluation of Free-Access Large Language Model Chatbots: A Multi-metric Comparative Study
摘要
Recently, the widespread adoption of Large Language Model (LLM) technologies has drawn significant attention. Both enterprises and individuals increasingly leverage these Artificial Intelligence (AI) systems across various services, particularly through chatbot interfaces. This study evaluates the current landscape of publicly accessible AI chatbots by comparing their performance and ability to assess the quality of their own generated content. We examine five widely used free-access chatbots: ChatGPT-3.5, Jamba, Phind, Perplexity, and Coral, using three question-answering datasets and one text summarization dataset. These chatbots are evaluated across nine metrics, four of which are assessed manually, while the remaining five employ AI-assisted evaluation using ChatGPT-3.5, Jamba, Microsoft Copilot, and Google Gemini. Our findings position ChatGPT-3.5 as the leading general-purpose model, a result that reflects its advanced development and maturity. AI-assisted metrics demonstrate strong performance across text comprehension and knowledge-based tasks, though results in mathematical reasoning tasks were inconsistent. Human analysis indicates that AI-assisted evaluations may inflate ratings, which reveals their potential bias favoring positive sentiment in current Large Language Models. This observed bias highlights well-documented limitations in text-based AI systems, warranting further research.