Anomaly Detection using Feature Selection

What is it about?

Anomaly pattern detection is very important in many domains, including cybersecurity. However, using the original dataset, anomaly detection is always a computationally expensive job. Moreover, the original dataset (often referred to as Big Data because of its characteristics, such as large volume, variety of data, and the speed of data generation) contains unnecessary data, for example, many redundant features or attributes. These irrelevant features may decrease the performance of machine learning algorithms. Removal of these irrelevant features and selection of only a suitable subset of relevant features from the original dataset can often increase the performance of machine learning algorithms. thereby, the performance of anomaly detection. Removal and selection of features require a search optimization technique and an evolutionary computation approach is generally used in this scenario. However, because of many limitations, such as when the search space is large, traditional evolutionary algorithms, for example, a genetic algorithm cannot meet the challenges. Hence, a variant of evolutionary algorithms called a cooperative co-evolution technique is most suitable in this case. The cooperative co-evolution decomposes a large problem into several subproblems, optimizes each subproblem individually, and collaborates the optimized subproblems at the end of each generation only to build a complete solution to the problem. This strategy helps maintain the proper population diversity and ensures appropriate evolutionary pressures, including selection, crossover, mutation. Following all these benefits, this paper proposed an anomaly detection approach using cooperative co-evolution-based feature selection in cybersecurity datasets.

Featured Image

Photo by Lewis Kang'ethe Ngugi on Unsplash

Photo by Lewis Kang'ethe Ngugi on Unsplash

Why is it important?

The proposed work was evaluated and validated in many ways, including taking a fixed number of examples from each to ensure effectiveness. The proposed work was experimented with and without a feature selection approach and also compared with the state-of-the-art anomaly detection techniques. Accordingly, the proposed approach can be used in many domains, such as cybersecurity and health sectors, where the regular anomalous activities cause a huge interruption to the business both financially and in terms of services. The proposed approach's execution time depends on the underlying Big Datasets. Hence, the underlying dataset can be preporcessed by removing the irrelevant features and selecting a suitable subset of only relevant features, the proposed anomaly detection approach will be one of the most effective approaches in the business.

Perspectives