Intelligent big data summarization for rare anomaly detection

Document Type

Journal Article

Publication Title

IEEE Access


Institute of Electrical and Electronics Engineers


ECU Security Research Institute / School of Science




Ahmed, M. (2019). Intelligent big data summarization for rare anomaly detection. IEEE Access, 7, 68669 - 68677. Available here


Identifying interesting patterns from a huge amount of data is a challenging task across a wide range of application domain. Especially, for cyber security being able to identify rare types of network activities or anomalies from network traffic data (a.k.a. Big Data!) is an important but time-consuming data analysis task having moderate computing resources. Existing research has shown that it is possible to detect rare anomalies from the summarized version of big data. Therefore, summarization is an effective preprocessing function before applying anomaly detection techniques. This aim of this paper is to improve and quantify the scalability and accuracy of the anomaly detection techniques by using summarization. Hence, we propose a sampling-based summarization technique (SUCh: Summarization Using Chernoff-Bound) which is computationally effective than the existing techniques and also performs better in identifying rare anomalies from twelve benchmark network traffic datasets. The experimental results show that, instead of using original dataset, a summary of the data yields better performance in terms of true positive and false positive rates, when used for anomaly detection with less time required.



Access Rights

subscription content

Research Themes

Securing Digital Futures

Priority Areas

Artificial intelligence and autonomous systems