Scholarlink

Abstract <p>A storage scheme is proposed for distributed machine learning on NeoFS (or any S3-compatible storage system), providing space efficiency and verifiable data integrity without the need for a consensus layer. Datasets are stored as erasure-coded shards with a single physical replica, and shard headers carry succinct integrity proofs. Clients verify shards by fetching only headers, then reconstruct samples by downloading any <i>k</i> shards in parallel. NeoFS storage policies handle placement, while an optional on-chain manifest anchors dataset commitments for auditability. We outline the shard format, header-only verification, and the end-to-end sampling pipeline. A LeNet-5 pilot shows that verification and decoding do not limit training throughput.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Decentralised Verifiable Coded Storage for Distributed Machine Learning

  • 期刊论文

  • 发表日期：

    2026年7月20日

查看全文
  • E. D. Pavlova

摘要

Abstract

A storage scheme is proposed for distributed machine learning on NeoFS (or any S3-compatible storage system), providing space efficiency and verifiable data integrity without the need for a consensus layer. Datasets are stored as erasure-coded shards with a single physical replica, and shard headers carry succinct integrity proofs. Clients verify shards by fetching only headers, then reconstruct samples by downloading any k shards in parallel. NeoFS storage policies handle placement, while an optional on-chain manifest anchors dataset commitments for auditability. We outline the shard format, header-only verification, and the end-to-end sampling pipeline. A LeNet-5 pilot shows that verification and decoding do not limit training throughput.