Scholarlink

Flagship large language models (LLMs), such as GPT-4 and its descendants, demonstrate exceptional capabilities in handling complex, multi-turn interactions in English. However, they come with high computational costs, making them impractical for widespread use. In contrast, relatively smaller models such as GPT-3.5 are significantly more cost-efficient but struggle with intricate tasks and maintaining context in multi-turn interactions. To realize high-quality multi-turn performance in cost-effective models, we enhance GPT-3.5 with a controller trained via reinforcement learning (RL). In our method, the agent consists of two components: a controller and an LLM. At each step, the controller selects an appropriate instruction from a predefined set based on the current observation, and the LLM’s input is formed by concatenating this instruction with the observation. The goal of the RL process is to train the controller to select the most appropriate instruction for each observation. We evaluate our method in two text-based environments, TextWorld and WebShop, where agents must interact over multiple turns to achieve complex goals. Experimental results show that our RL-Enhanced GPT-3.5 achieves performance comparable to GPT-4 while maintaining the cost-efficiency of GPT-3.5.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Enhancing Cost-Effective Large Language Models with Reinforcement Learning for Multi-turn Text-Based Environments

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Hanhua Zhu,
  • Tomoyuki Kaneko

摘要

Flagship large language models (LLMs), such as GPT-4 and its descendants, demonstrate exceptional capabilities in handling complex, multi-turn interactions in English. However, they come with high computational costs, making them impractical for widespread use. In contrast, relatively smaller models such as GPT-3.5 are significantly more cost-efficient but struggle with intricate tasks and maintaining context in multi-turn interactions. To realize high-quality multi-turn performance in cost-effective models, we enhance GPT-3.5 with a controller trained via reinforcement learning (RL). In our method, the agent consists of two components: a controller and an LLM. At each step, the controller selects an appropriate instruction from a predefined set based on the current observation, and the LLM’s input is formed by concatenating this instruction with the observation. The goal of the RL process is to train the controller to select the most appropriate instruction for each observation. We evaluate our method in two text-based environments, TextWorld and WebShop, where agents must interact over multiple turns to achieve complex goals. Experimental results show that our RL-Enhanced GPT-3.5 achieves performance comparable to GPT-4 while maintaining the cost-efficiency of GPT-3.5.