An Enhanced Ensemble Machine Learning Framework with Explainable AI for Cyber Threat and Network Outlier Detection

Abstract

Intrusion detection systems (IDS) are a cornerstone of modern cybersecurity infrastructure. Traditional machine learning approaches for IDS—including artificial neural networks (ANNs)—often suffer from class imbalance, limited generalization, and insufficient interpretability. This paper presents an enhanced ensemble machine learning framework that integrates Random Forest (RF), XGBoost, LightGBM, and a Multi-Layer Perceptron (MLP) deep learning classifier with a comprehensive preprocessing pipeline on the CICIDS2017 benchmark dataset. The preprocessing pipeline incorporates duplicate removal, IQR-based outlier clipping, Yeo-Johnson power transformation, standard scaling, and Principal Component Analysis (PCA) retaining 95% of explained variance. Experimental results demonstrate that LightGBM achieves the highest accuracy of 99.81% with an AUC-ROC of 0.9998, Matthews Correlation Coefficient (MCC) of 0.9832, and a training time of only 4.7 seconds. To address model transparency, Local Interpretable Model-agnostic Explanations (LIME) are applied to the best model to provide feature-level decision explanations. Comparative evaluation against a baseline ANN (92% accuracy) and prior state-of-the-art methods confirms the superiority of the proposed framework. These results demonstrate that ensemble methods with principled preprocessing and explainability mechanisms can significantly advance the effectiveness and trustworthiness of cyber threat detection.