The paper presents TopicMapper, a tool designed to extract and categorize topics from geotagged URLs, focusing on web pages within specific geographical regions. The tool utilizes Apache Spark for large-scale data processing and employs the Latent Dirichlet Allocation (LDA) algorithm to identify topics within web pages. The system architecture involves parallelization and a multi-step pipeline, enabling efficient data handling and workload optimization. The study demonstrates the scalability of the tool through experiments on datasets of varying sizes. The results reveal that TopicMapper exhibits promising scalability, maintaining quasilinear performance as dataset sizes increase. The paper also introduces a method to avoid unnecessary recomputation by considering common areas between previously computed and new regions. Future work may involve GUI development for user-friendly interaction and refinement of the recomputation avoidance technique.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

TopicMapper: Geotagged URL Topic Extraction and Categorization

  • Vijay Kumar

摘要

The paper presents TopicMapper, a tool designed to extract and categorize topics from geotagged URLs, focusing on web pages within specific geographical regions. The tool utilizes Apache Spark for large-scale data processing and employs the Latent Dirichlet Allocation (LDA) algorithm to identify topics within web pages. The system architecture involves parallelization and a multi-step pipeline, enabling efficient data handling and workload optimization. The study demonstrates the scalability of the tool through experiments on datasets of varying sizes. The results reveal that TopicMapper exhibits promising scalability, maintaining quasilinear performance as dataset sizes increase. The paper also introduces a method to avoid unnecessary recomputation by considering common areas between previously computed and new regions. Future work may involve GUI development for user-friendly interaction and refinement of the recomputation avoidance technique.