In the era of big data, the efficiency of the Hyperlink-Induced Topic Search (HITS) algorithm in processing large-scale web link data has become a critical issue. To address the inefficiency of the traditional serial HITS algorithm when dealing with massive web link graphs, this paper proposes a novel parallel HITS algorithm. The algorithm lies in the rational partitioning of the web link graph. By adopting a graph-based partitioning method, the large-scale web link graph is divided into multiple subgraphs. Each subgraph is assigned to a computing node, enabling each node to independently calculate the authority and hub values of the web pages within its assigned subgraph. In addition, to reduce communication overhead in the parallel computing process, we further design data compression and asynchronous communication strategies. The former is applied to web link data before transmission to effectively reduce the amount of data transferred, while the latter enables processing units to perform other tasks while waiting for data transmission, thereby improving resource utilization. Experimental results demonstrate that the proposed parallel HITS algorithm not only maintains the accuracy of the original HITS algorithm but also achieves a significant improvement in computing efficiency.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

PHITS: A Parallel Hyperlink-Induced Topic Search Algorithm with Graph Partitioning and Communication Optimization

  • Xuanye Chen,
  • Xiaoshuang Xing,
  • Mengjiao Ou,
  • Jialin Chen,
  • Xiaoyu Ma

摘要

In the era of big data, the efficiency of the Hyperlink-Induced Topic Search (HITS) algorithm in processing large-scale web link data has become a critical issue. To address the inefficiency of the traditional serial HITS algorithm when dealing with massive web link graphs, this paper proposes a novel parallel HITS algorithm. The algorithm lies in the rational partitioning of the web link graph. By adopting a graph-based partitioning method, the large-scale web link graph is divided into multiple subgraphs. Each subgraph is assigned to a computing node, enabling each node to independently calculate the authority and hub values of the web pages within its assigned subgraph. In addition, to reduce communication overhead in the parallel computing process, we further design data compression and asynchronous communication strategies. The former is applied to web link data before transmission to effectively reduce the amount of data transferred, while the latter enables processing units to perform other tasks while waiting for data transmission, thereby improving resource utilization. Experimental results demonstrate that the proposed parallel HITS algorithm not only maintains the accuracy of the original HITS algorithm but also achieves a significant improvement in computing efficiency.