Leveraging Blockchain for Decentralized Training and Inference of Large Language Models
摘要
The substantial computational resources and expertise required to train and run AI models pose significant barriers of entry to the AI market. For instance, GPT-3 took USD 250 million to train, limiting participation to well-funded organizations and research institutions. This paper explores the integration of blockchain technology as a decentralized framework to distribute the training and inference processes of LLMs, while also focusing on AI model architecture that enables these adaptations. Previous work done by PETALS [2] and learning-at-home [10] has made this possible throughout different advancements in AI inferencing and training; However, the networks are not sustainable due to the user interactions needed to choose which model layer to host, without insights into the network health. This creates lack of coordination between nodes and results in an imbalance of model layer distribution in the network. We propose a blockchain-based framework that enables individuals and organizations to contribute computational resources and data for training and inference tasks in exchange for incentives. This decentralized approach leverages smart contracts for task allocation, verification, and reward distribution, ensuring transparency and traceability throughout the contribution of the nodes. Once the blockchain is populated with data, aggregated reports generated by network nodes are used to score the network and its nodes in order to find the most suitable model layers to host for inferencing, or experts to train for new nodes. The paper addresses key challenges such as efficient task distribution, model architecture, and maintaining a sustainable AI network. We also discuss how blockchain’s inherent properties can be harnessed to create a trustworthy and scalable platform for AI development.