The emergence of LLMs as a powerful tool for generating human-like texts has raised concerns with severe sociological and ethical, and cyber effects in almost all domains. In particular, the unregulated use of these models in an academic setting has serious consequences, including academic dishonesty, deception, copyright infringement, plagiarism, misinformation dissemination, erosion of critical thinking, factual inconsistency, and unreliability. Hence, to ensure responsible use of LLMs, it is necessary to have a reliable system for the detection of AI-generated text. This work investigates the use of various machine learning, deep learning, and LLM models for detecting AI-generated text. The problem is implemented as a supervised binary classification problem with models trained on human-written text as well as AI-generated text. The methodology adapted is to train different models on benchmark data sets to find the best suited model and then apply the model to unseen dataset. In all performance analysis of 25 models (5 five deep learning models, 15 machine models and 4 LLM models) were compared. The models were trained on 6 different benchmark datasets, each with at least 200,000 text inputs in the form of paragraphs. The variation of performance of the model on different datasets, both balanced and imbalanced datasets have been used. Amongst all the models, DistilBERT was found to provide the most optimal output with an accuracy of 95% and 93% on training and validation data respectively. To illustrate the utility of this optimal model in the educational environment, the unseen test data was prepared manually by extracting 500 paragraphs from NCERT standard 12th English textbook labelled as human written text. AI-generated content was produced for the same paragraphs using ChatGPT, Gemini, Perplexity.ai and Claudia and were labelled as AI generated content. The paragraphs in test data were entirely different in context from the train data. On applying DistilBERT on this dataset, it was found to detect AI generated text well with an accuracy of 96% even when the unseen data is from different context establishing its robustness.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Analyzing AI-Generated Text Using Machine Learning, Deep Learning and Large Language Models

  • Sneha Arun,
  • Suchit Purohit,
  • Jyoti Pareek

摘要

The emergence of LLMs as a powerful tool for generating human-like texts has raised concerns with severe sociological and ethical, and cyber effects in almost all domains. In particular, the unregulated use of these models in an academic setting has serious consequences, including academic dishonesty, deception, copyright infringement, plagiarism, misinformation dissemination, erosion of critical thinking, factual inconsistency, and unreliability. Hence, to ensure responsible use of LLMs, it is necessary to have a reliable system for the detection of AI-generated text. This work investigates the use of various machine learning, deep learning, and LLM models for detecting AI-generated text. The problem is implemented as a supervised binary classification problem with models trained on human-written text as well as AI-generated text. The methodology adapted is to train different models on benchmark data sets to find the best suited model and then apply the model to unseen dataset. In all performance analysis of 25 models (5 five deep learning models, 15 machine models and 4 LLM models) were compared. The models were trained on 6 different benchmark datasets, each with at least 200,000 text inputs in the form of paragraphs. The variation of performance of the model on different datasets, both balanced and imbalanced datasets have been used. Amongst all the models, DistilBERT was found to provide the most optimal output with an accuracy of 95% and 93% on training and validation data respectively. To illustrate the utility of this optimal model in the educational environment, the unseen test data was prepared manually by extracting 500 paragraphs from NCERT standard 12th English textbook labelled as human written text. AI-generated content was produced for the same paragraphs using ChatGPT, Gemini, Perplexity.ai and Claudia and were labelled as AI generated content. The paragraphs in test data were entirely different in context from the train data. On applying DistilBERT on this dataset, it was found to detect AI generated text well with an accuracy of 96% even when the unseen data is from different context establishing its robustness.