Evaluating scientific publications is a crucial tool for understanding their impact and scientific significance. However, traditional metrics such as Impact Factor (IF), h-index and citation count face new challenges with the development of Artificial Intelligence (AI). In this article, we evaluate the usefulness of existing measures and discuss the need for a new AI metric that more accurately reflects the content and significance of the scientific literature. We review traditional metrics for assessing the quality of research and point out their shortcomings in the context of recent developments. We also provide an overview of existing approaches that use AI techniques to assess the quality and significance of scientific papers, along with examples of models and algorithms used. We provide a new AI-based measure that explains the desired criteria and development process for assessing scientific papers. We also provide case studies of this new measure’s implementation and contrast its outcomes with those of other contemporary techniques and conventional measurements. Lastly, we go over possible difficulties and restrictions in using this new measure as well as moral and practical arguments in favor of its use. The article’s conclusion includes a summary of the key ideas, a focus on the necessity of creating new AI-based metrics to enhance the assessment of scientific publications, as well as a perspective and suggestions for scholars, editors, and decision-makers working in this area.