This paper highlights the relevance of applying Natural Language Processing (NLP) and Data Cleaning techniques to input prompts and their direct impact on reducing costs associated with the use of Large Language Models (LLMs). It analyzes how implementing fundamental preprocessing tasks in input prompts can reduce costs by up to 65% concerning the handling of tokens processed from user prompts, achieving up to 100% reliability from the LLM in some cases. The research was tested on gpt-4o-mini, gpt-4, and gpt-3.5-turbo models. Additionally, key performance indicators (KPIs) are included, along with their interpretation. This paper does not propose new preprocessing or text tokenization techniques; instead, it introduces a strategy called “Sequential Text Compression Strategy” to maintain the quality of outputs while optimizing resources. Finally, the paper discusses a new term called “Text-to-Token Compression Paradox”, which refers to the lack of a coherent relationship between the number of characters in a text and the number of generated tokens.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Sequential Text Compression Strategy: A Study of NLP and Data Cleaning Preprocessing Tasks for Optimizing Token Utilization in LLMs

  • Erick Berssaín García V.

摘要

This paper highlights the relevance of applying Natural Language Processing (NLP) and Data Cleaning techniques to input prompts and their direct impact on reducing costs associated with the use of Large Language Models (LLMs). It analyzes how implementing fundamental preprocessing tasks in input prompts can reduce costs by up to 65% concerning the handling of tokens processed from user prompts, achieving up to 100% reliability from the LLM in some cases. The research was tested on gpt-4o-mini, gpt-4, and gpt-3.5-turbo models. Additionally, key performance indicators (KPIs) are included, along with their interpretation. This paper does not propose new preprocessing or text tokenization techniques; instead, it introduces a strategy called “Sequential Text Compression Strategy” to maintain the quality of outputs while optimizing resources. Finally, the paper discusses a new term called “Text-to-Token Compression Paradox”, which refers to the lack of a coherent relationship between the number of characters in a text and the number of generated tokens.