Public sentiment on social media offers valuable insights into economic trends; however, real-time analysis of recession-related tweets using Spark Structured Streaming remains relatively unexplored. Governments require such analytics to assess public distress, predict economic behavior, and implement timely interventions. This study aims to bridge that gap by analyzing a synthetic dataset that mimics Indian Twitter reactions to the 2023 recession. Employing Apache Spark, we developed a scalable pipeline to classify tweets based on sentiment, sarcasm, depressive tone, sector relevance, and suggestions. The dataset is divided into an 80:20 ratio for training and evaluation, and machine learning models are trained using Spark MLlib. The Random Forest model achieved an impressive 98.88% accuracy in sentiment classification, 99.99% in suggesting recommendations, and 90.00% in depression detection model; XGBoost recorded 100% in sarcasm detection; and Naive Bayes reached 84.49% accuracy in sector classification, surpassing the 80.00% performance threshold. A real-time Flask dashboard visualizes these predictions through dynamic charts, facilitating immediate policy responses. This work illustrates how Spark Structured Streaming can effectively process live social media data on a scale, while providing governments with a proactive tool to monitor sentiment during recession and formulate data-driven strategies, hence resulting in better decisions that are more considerate of people’s needs.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Spark-Based Pipeline for Real-Time Multi-Faceted Analysis of Public Sentiment During the Indian Recession

  • Deshana Vikas Shah,
  • S. Saravanan

摘要

Public sentiment on social media offers valuable insights into economic trends; however, real-time analysis of recession-related tweets using Spark Structured Streaming remains relatively unexplored. Governments require such analytics to assess public distress, predict economic behavior, and implement timely interventions. This study aims to bridge that gap by analyzing a synthetic dataset that mimics Indian Twitter reactions to the 2023 recession. Employing Apache Spark, we developed a scalable pipeline to classify tweets based on sentiment, sarcasm, depressive tone, sector relevance, and suggestions. The dataset is divided into an 80:20 ratio for training and evaluation, and machine learning models are trained using Spark MLlib. The Random Forest model achieved an impressive 98.88% accuracy in sentiment classification, 99.99% in suggesting recommendations, and 90.00% in depression detection model; XGBoost recorded 100% in sarcasm detection; and Naive Bayes reached 84.49% accuracy in sector classification, surpassing the 80.00% performance threshold. A real-time Flask dashboard visualizes these predictions through dynamic charts, facilitating immediate policy responses. This work illustrates how Spark Structured Streaming can effectively process live social media data on a scale, while providing governments with a proactive tool to monitor sentiment during recession and formulate data-driven strategies, hence resulting in better decisions that are more considerate of people’s needs.