An approach to behaviour-based intrusion detection system : implementation of a role-based network access control system and a proposal for three detection strategies

Date of Award


Degree Type


Degree Name

Master of Science


School of Computer and Information Science


Faculty of Computing, Health and Science


The number of computers connecting with the Internet has grown rapidly, and user numbers are burgeoning, consequently server numbers are increasing proportionately in response to this new demand for services. An increasing population of vulnerable machines has been created with numbers of so-called “malicious users” also escalating dramatically resulting in millions of Internet users now suffer from Virus and Trojan horse interference.

In recent years, many security measures have been created. Firewalls are a typical as they play the role of network checkpoint, providing a secured means of regulating the outgoing and incoming information of the network’s area. An intrusion detection system (IDS) is an active process to observe attacks before a targeted system incurs fatal damages. Detecting intrusion can be categorized into two groups according to the method used: knowledge-based and behaviour-based. Knowledgebased detection techniques make use of all the knowledge pertaining to specific attacks, while behaviour-based detection is knowledgeable about normal communication, being able to detect intrusions by scrutinizing deviations sifted from expected or normal traffic. Currently, most IDS are classified as knowledge-based, with very few behaviour-based systems in use. The “Behaviour” is something associated with ’Who‘ – whose behaviour or to whom the behaviour applies; ’When‘ – when does a behaviour happen; ’What‘ – what happens when a behaviour occurs, or for what purpose, and so on. This requires a detection engine that has an unusually good knowledge of intranet infrastructure and protocols to guard against intrusions.

This study attempts to develop feasible solutions and provide practical approaches to implement a behaviour-based intrusion detection, based on the Netfilter implementation. This is a default firewall framework in Linux kernel, comprising statistical anomaly detection techniques and artificial intelligence. The Linux default stateful firewall netfilter not only filters the content of each package, but summarizes connection information into states, it has already opened the possibility of a behaviour-based IDS. This research develops a proposed detection engine that improves the performance of detection compared to netfilter. The proposed IDS in this thesis first analyses audit data collected from the network data sensor, relying on netfilter in such a way that each connection is endowed with a role attribute, then performing real-time monitoring on activities that may be unusual. This analysis is performed using statistical analysis, association rule data mining, and an expert system. The engine can therefore handle protocol-based inspection, and analyse internal network users’ networking behaviours.

To this end, this project has enabled a role-based network access control subsystem to be implemented into netfilter. Furthermore, three detection strategies have been proposed to apply in three different situations: weight-based statistical analysis is applied to TCP state transition analysis; association rule data mining is proposed for learning correlation of HTTP connection events; and an expert system is introduced to evaluate the behaviour of the connection, employing a knowledge base (a set of rules) to reach conclusions. All techniques are presented for gradually achieving a high level of intelligence in the detection of anomaly. This study contributes to the process of modifying the network implementation of a Linux operating system, and to its incorporation using artificial intelligence techniques into network security via a behaviour-based detection approach to an Intrusion Detection System.

LCSH Subject Headings

Intrusion detection systems (Computer security)

Computer security.

Computer networks -- Security measures -- Evaluation.

Access Note

Access to this thesis - the full text is restricted to current ECU staff and students by author's request. Email request to library@ecu.edu.au

Access to this thesis is restricted. Please see the Access Note below for access details.