A Machine Learning-Based Early Warning System for Student Performance Prediction: System Development and Empirical Evaluation in Higher Education

Abstract

Academic failure and student attrition remain structural challenges in higher education, yet the behavioral and academic data generated through digital learning environments are largely underexploited for early risk identification. This study presents the design, development, and empirical evaluation of a machine learning-based early warning system (EWS) for predicting student academic performance, deployed as a fully operational web-based application within the information technology infrastructure of a higher education institution in Indonesia. A dataset of 1,240 student records spanning academic years 2021–2024 was constructed by integrating static academic attributes extracted from the institutional Academic Information System (SIAKAD) with dynamic behavioral features derived from a Moodle-based Learning Management System (LMS), including weekly login frequency, assignment submission lead time, quiz attempt rate, and forum participation count. Four supervised classification algorithms Logistic Regression, Random Forest, Gradient Boosting, and Long Short-Term Memory (LSTM) were trained and benchmarked under stratified 10-fold cross-validation with SMOTE-based class balancing. The Gradient Boosting classifier achieved superior performance across all evaluation metrics, attaining an accuracy of 89.1%, F1-Score of 0.850, and AUC-ROC of 0.931. SHAP-based feature attribution confirmed that LMS-derived behavioral variables, particularly weekly login frequency (SHAP = 0.241) and assignment submission lead time (SHAP = 0.187), contributed substantively to prediction quality beyond static academic records alone. The deployed system was evaluated by 32 academic advisors using the System Usability Scale (SUS) following a four-week observation period, yielding a mean score of 78.4, indicative of above-average practitioner usability.