Scholarlink

The performance of Large Language Models (LLMs) relies on massive and high-quality pre-processed datasets, often exceeding several terabytes in size. Maintaining such large datasets is challenging for a single organization. A collaborative framework between multiple organizations would be required. We propose a distributed dataset framework that separates data pre-processing and LLM pre-training, enabling an ML pipeline over wide area networks. Kogitune is a proposed framework that optimizes LLM pre-training by mixing pre-processed datasets from multiple remote storage sources. This paper presents the design and a prototype implementation of Kogitune and discusses our experiences using Kogitune to develop small Japanese code LLMs. We release Kogitune as open-source software on GitHub.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Distributed Dataset Framework for Large Language Models Pre-training

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Nao Souma,
  • Yui Obara,
  • Yasuhiko Yokote,
  • Yutaka Ishikawa,
  • Kimio Kuramitsu

摘要

The performance of Large Language Models (LLMs) relies on massive and high-quality pre-processed datasets, often exceeding several terabytes in size. Maintaining such large datasets is challenging for a single organization. A collaborative framework between multiple organizations would be required. We propose a distributed dataset framework that separates data pre-processing and LLM pre-training, enabling an ML pipeline over wide area networks. Kogitune is a proposed framework that optimizes LLM pre-training by mixing pre-processed datasets from multiple remote storage sources. This paper presents the design and a prototype implementation of Kogitune and discusses our experiences using Kogitune to develop small Japanese code LLMs. We release Kogitune as open-source software on GitHub.