Scholarlink

<p>In this paper, we introduce a scalable distributed local learning algorithms for large-scale multi-class classification implemented on Apache Spark. Unlike conventional approaches that rely on a single global non-linear model, our method decomposes the learning task into a collection of lightweight local models, constructed from structurally simpler regions of the training data. This design is motivated by the hypothesis that local decision boundaries are significantly less complex and therefore easier to approximate than global ones. We propose two complementary local-learning strategies. The first is a pre-partitioned approach, where the training data are clustered using <i>k</i>-Means and local models are trained in parallel within each cluster. The second is an adaptive test-time approach, where a local model is dynamically constructed from the global <i>k</i>-nearest neighbors (<i>k</i>NN) of each query instance. Both strategies are fully parallelized on Apache Spark, ensuring efficient distributed computation and scalability to million-scale datasets. We conduct extensive experiments on the ImageNet dataset (1,281,167 images, 1000 classes) using pre-extracted ViT-Base/16 embeddings. In a two-PC distributed environment, the <i>k</i>-Means-based variant achieves 89.60% Top-1 accuracy in 6.33&#xa0;min, while the adaptive <i>k</i>NN-based variant attains 90.00% Top-1 accuracy in 19.42&#xa0;min. These results demonstrate that local learning remains effective even at the scale of ImageNet, while offering a flexible trade-off between computational efficiency and predictive performance, establishing it as a competitive alternative to global non-linear models for large-scale classification.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Distributed Local Learning on Apache Spark for Handling Large Datasets

  • 期刊论文

  • 发表日期：

    2026年4月13日

查看全文
  • Quoc-Bao Bui-Vo,
  • Thanh-Nghi Do

摘要

In this paper, we introduce a scalable distributed local learning algorithms for large-scale multi-class classification implemented on Apache Spark. Unlike conventional approaches that rely on a single global non-linear model, our method decomposes the learning task into a collection of lightweight local models, constructed from structurally simpler regions of the training data. This design is motivated by the hypothesis that local decision boundaries are significantly less complex and therefore easier to approximate than global ones. We propose two complementary local-learning strategies. The first is a pre-partitioned approach, where the training data are clustered using k-Means and local models are trained in parallel within each cluster. The second is an adaptive test-time approach, where a local model is dynamically constructed from the global k-nearest neighbors (kNN) of each query instance. Both strategies are fully parallelized on Apache Spark, ensuring efficient distributed computation and scalability to million-scale datasets. We conduct extensive experiments on the ImageNet dataset (1,281,167 images, 1000 classes) using pre-extracted ViT-Base/16 embeddings. In a two-PC distributed environment, the k-Means-based variant achieves 89.60% Top-1 accuracy in 6.33 min, while the adaptive kNN-based variant attains 90.00% Top-1 accuracy in 19.42 min. These results demonstrate that local learning remains effective even at the scale of ImageNet, while offering a flexible trade-off between computational efficiency and predictive performance, establishing it as a competitive alternative to global non-linear models for large-scale classification.