Scalable Big Data Architecture: Improving Data Management at MoHE
摘要
Higher education institutions generate large amounts of data from various sources, including student information systems, learning management systems, administrative records, etc. For this reason, systematic data management and use are essential in decision-making and strategic planning. However, data management practices are currently struggling to handle the growing volume and complexity of data, making it challenging to obtain useful information for quick decision-making. Furthermore, integrating heterogeneous data sources and processing data in real-time while maintaining data quality remains challenging. Therefore, in the context of Malaysian education, this study proposes a large-scale big-data architecture designed specifically for the Malaysian Ministry of Higher Education (MoHE). It uses Apache technologies such as Apache Hadoop, Apache Spark, Apache Kafka, Apache NiFi, and Apache Cassandra to provide comprehensive solutions for data integration, real-time processing, and data quality management. This architecture aims to improve the efficiency and effectiveness of higher education data management in making better decisions. This innovative approach enable MoHE to can make more accurate decisions based on multiple data sources, enable real-time analysis, and ensure robust data governance, leading to a more responsive and data-driven educational environment.