Cải thiện phân lớp email rác bằng cách kết hợp các giải thuật lựa chọn

IMPROVING SPAM CLASSIFICATION BY

MEANS OF PROTOTYPE SELECTION ENSEMBLES

CẢI THIỆN PHÂN LỚP EMAIL RÁC BẰNG CÁCH

KẾT HỢP CÁC GIẢI THUẬT LỰA CHỌN

MSc. Mai Thu Giang

University of Economics, Hue University

Abstract

Spam mail is one of the disadvantages of e-mail application. This problem annoys

recipient, taking their time, as well as wasting network bandwidth. From this reason,

SPAM classification is a realistic problem which we need to consider. Furthermore,

modeling a system which automatically classifies spam email from our list of emails has

become important. In order to build this system more effectively, one of the methods we

often use in the pre-processing procedure is Prototype selection (PS), which decides

removing or keeping instances during generalization. This study will work with three state-

of-the-art PS algorithms, including Extended Nearest Neighbors (ENN), Decremental

Reduction Optimization procedure 3 (DROP3), Random Mutation Hill Climbing (RMHC),

and the method of combination these algorithms into one ensemble method. The reduction

of storing requirements, speed of computation and the accuracy of classifications based on

the KNN method is tested carefully. These results confirm that in some cases, the combined

method leads to higher accuracy of classification as well as lower storage requirement

than each individual method, in which the combination between ENN and DROP3 result

the highest accuracy.

Keywords: Extended Nearest Neighbors (ENN), Decremental Reduction Optimization

procedure 3 (DROP3), Random Mutation Hill Climbing (RMHC), kNearest Neighbors

(KNN).

Tóm tắt

Thư rác (SPAM mail) là một trong những nhược điểm của ứng dụng thư điện tử.

Vấn đề này gây ra nhiều phiền toái cho người nhận, làm mất thời gian của người nhận

cũng như lãng phí băng thông mạng. Từ lý do này, phân loại thư rác là một vấn đề thực tế

mà chúng ta cần xem xét. Hơn nữa, việc mô hình hóa một hệ thống tự động phân loại thư

rác từ danh sách thư càng ngày càng trở nên quan trọng. Điều đó cần có những giải thuật

mang lại kết quả phân lớp cao để áp dụng vào mô hình. Để xây dựng hệ thống này hiệu

quả hơn, một trong những phương pháp thường được sử dụng trong quy trình tiền xử lý là

lựa chọn nguyên mẫu (Prototype Selection), quyết định loại bỏ hoặc giữ các mẫu trong

quá trình khái quát hóa. Nghiên cứu này sẽ làm việc với ba phương pháp, bao gồm ENN

(Mở rộng lân cận), DROP3 (Tối ưu hóa giảm dần), RMHC (Leo đồi đột biến ngẫu nhiên)

và phương pháp kết hợp các thuật toán này. Việc giảm các yêu cầu lưu trữ, tăng tốc độ

tính toán và tăng độ chính xác của phân loại dựa trên phương pháp K hàng xóm lân cận

(KNN) được kiểm tra cẩn thận. Kết quả của nghiên cứu cho thấy rằng, trong một số

839

trường hợp, việc kết hợp các phương pháp cho độ chính xác cao hơn và yêu cầu lưu trữ

thấp hơn so với từng phương pháp riêng lẻ. Trong đó, sự kết hợp giữa ENN và DROP3

mang lại độ chính xác cao nhất.

Từ khóa: ENN (Mở rộng lân cận), DROP3 (Tối ưu hóa giảm dần), RMHC (Leo đồi đột

biến ngẫu nhiên).

1. Introduction

In the list of emails which we receive everyday, there are a huge number of

unexpected emails which are called spam mails. These emails take us time to remove them

from our list of emails as well as occupy a large space bandwidth. We need a system,

which can help us to classify or recognize an unknown email, is spam mail or ham mail

before allowed just ham mails can come to our inbox. In order to solve this problem, we

have to work with the classification methods. The classification methods aim to predict the

class of new instance based on the knowledge in the given the training data T. there are

many classification methods such as k nearest neighbors (kNN), decision tree (DT),

support vector machine (SVM). However, in this study, we focus on using baseline method

kNN. kNN looks for the k nearest neighbors (NNs) of the given instance in the set of

prototypes. The result will be the class that most of these neighbors belong to.

In order to classify effectively, the pre-processing dataset of mails is very important

step which can improve the accuracy of the classification, reduce storage requirements, and

improve speed of classification. One of the solution for mentioned pre-processing step is

Prototype Selection, which selects a subset S of the training set T, and a new instance is

classified based on the subset S instead of original training set T.

It is clear that there are many studies related to PS problems (Aggarwal & Amrita,

2013). However, there is no research on the combination of new training sets which result

from separated algorithms and assessing the effect of these combinations. This study works

with three algorithms: ENN, DROP3, and RMHC. In which, ENN is the most recent

supervised learning algorithm for pattern recognition (Bo & Haibo, 2015); DROP3 is the

optimal algorithm in decrimental reduction features (Wilson & Martinez, 2000); and

RMHC is the search method applying genetic algorithm to get the highest fitness of

population (Lu, Jie, Wuyang, & Shasha, 2017).

The details of each algorithm will be presented in the following part of the paper.

The study solves the problem in the following steps:

(1) apply kNN classification to the original dataset.

(2) Apply kNN classification to three new training sets which obtained from ENN,

DROP3, and RMHC.

(3) Access the improvement respect to accuracy, storage requirements, and time

cost.

(4) The last step and also the most important step in this study is getting the

intersection between each pair of new training sets from step 2, as well as the intersection

840

of three new training sets, after that using kNN classification for these intersection sets and

accessing the effect of these combinations.

The results of the study show that in some situation, the combined method has

more effects on getting better classification than each original method.

2. Literature review and Methods

2.1. Literature review

In this section we will briefly present research related to the study:

2.1.1 The k-Nearest Neithbor (kNN) algorithms

The 1-nearest neighbor (1-NN) classification algorithm is used in our study. This

algorithm conducted by Wilson and Martinez (Wilson & Martinez, 2000). The similarity of

two instances is accessed through the similarity distance between two scaled instances, by

using Euclidian distance function relies on weighted features.

The distance between two scaled instances by using the Euclidian distance function is

defined as:

Where and are the two input vectors of two instances with m features, m is the

number of input features, and x_iand y_iare the input values for input feature i (Wilson &

Martinez, 2000).

The instance with the smallest distance to a test instance is its 1-nearest neighbor.

The instance is correctly classified by 1-NN when its class label is as similar as the test

instance.

2.1.2 The Prototype Selection

There are three types of techniques for reducing instance, inluding condensation,

edition and hybrid techniques.

Firstly, condensation techniques improve running time and reduce storage

requirements. One of the most popular algorithms for these techniques designed for 1NN is

Condensed Nearest Neighbors (Nele & Cornelis, 2013). In this algorithm, with the original

training set T, the subset S is initialized with one instance from each class by randomly.

After that classify each instance x in the T using S as training set by apply 1NN rule. If x is

misclassified then adding x to S, otherwise we do not need to consider this instance. The

new training set which we get after applying this algorithm is S. Obviously, all instances in

the new training set by using 1NN is classified correctly over S. Some other algorithms in

this technique is Modified Condensed Nearest Neighbor (MCNN), Reduced Nearest

Neighbor (RNN),...

Secondly, edition techniques try to reduce the size of the original training set T, but

to improve classification quality of the kNN rule by removing noisy instances (Nele &

841

Cornelis, 2013). For examples, Edited Nearest Neighbor (ENN), Repeated-ENN (RENN),

All k-NN (AllKNN), Fuzzy Rough Prototype Selection (FRPS),...are popular algorithms,

which belong to edition techniques.

Thirdly, hybrid techniques are the combination of edition and condensation

techniques (John & Eliot, 2006). These techniques have both advantages from edition and

condensation techniques. On the one side, these techniques are designed to reduce as much

as possible the dimension space of the dataset, or reduce store requirements. On the other

side, they try to improve the accuracy of classification using the kNN rule. Some typical

algorithms for the techniques are Random Mutation Hill Climbing (RMHC), Decremental

reduction optimization procedure 1-3 (DROP1-DROP3), Generational Genetic Algorithm

(GGA).

2.2.2 Direction of search

There are three directions can be proceed during searching for a subset of training

set S from original training set T, including incremental, decremental, and batch (Shi,

Wang, & Weng, 2012).

In terms of incremental search direction, at the fist step, the subset of training set S

is initialized with an empty. After that each instance from T is added to S if it satisfies

given condition. In this case the order of presentation of instances can be very important

because it has effect on deciding which instances are considered first, leading to adding

their neighbors to the subset S or not.

On the other hand, the decremental search begins with the subset S is initialized by

the original training set T. during the search, each instance is considered, and it is removed

from the subset S when it satisfies given condition. It means that the number of instance in

this search direction will decrease during the time of searching. As the same as incremental

search direction, the order of instance in the dataset have effects on deciding which

instance is removed first, leading to effects on its neighbors. However, the decremental

search is more expensive than incremental algorithms (Nele & Cornelis, 2013).

Last but not least, the batch mode involves deciding if each instance meets the

removal criteria before removing any of them. After that all of instances which satisfy

given condition of removing will be removed at once.

2.2.3 Border points vs. central points

One of the parts of knowledge which directly related the algorithms which we will

discuss later is border points and central points.

The central points are “internal” points, which do not have effect on decision the

boundaries, while the border points decide which is boundary of a class. Because of these,

we can see that the central points can be removed with weak effects on the classification.

However, in some other cases, the border points, which are noisy or do not agree with their

neighbors, are removed. This action will remove close border points, resulting smoother

decision boundaries (Bien & Tibshirani, 2012).

842

2.1.3. The Algorithms

2.1.3.1 Edited Nearest Neighbor (ENN)

At the first step procedure, the subset training S is initialized by the original

training set T. After that, classify each instance x in S using S\{x} as training set by using

1NN classification. If x is misclassified then remove it from the subset S. The result subset

S contains every instance classified correctly over 1 NN rule.

The evaluation of search for ENN is filter techniques. The direction of search for

ENN is decremental because the subset S is initialized by the whole training set, after that

each instance is removed from S if it is misclassified.

The pseudocode for ENN is given as follows (Alejo, Sotoca, & Valdovinos, 2010):

T original training set

1. S = T

2. Classify each instance x in S using S\{x} as training set (KNN classification,

k=1)

1. x misclassified -> remove x from S

If |S|=3 -> go to 3.

2. x correctly classified -> do nothing

3. Return S

For example, there is a training set T includes 7 instances, which belong to two

classes (-) and (+).

S\{X1}

S=T

X3(-)

X2(-)

X4(+)

X1(+)

X5(+)

X6(+)

X7(+)

Figure 1. Example for classification the training set of 7 instances by using ENN,

with two classes (+) and (-)

At the first step, the subset S is initialized by the training set T. In order to classify

for X1 by using 1NN rule and using S\ {X1} as the training set. Then we can see that X2 is

the nearest neighbor of X1. However, X1 is misclassified because X1’s class is (+) while

X2’s class is (-) then after this step, X1 is removed from S. According to the algorithm, in

the next steps, X2, X3 will be removed from S. The result subset S = {X4, X5, X6, X7}.

2.1.3.2 Decremental Reduction Optimization Procedure (DROP1-3)

DROP1-3 is one of the new heuristic algorithms in the list of PS algorithms. These

algorithms not only consider carefully the similarity distance of instances and the

misclassification based on kNN rule, but also take into account the order of which

instances are removed in the next step.

843

In order to work with DROP3, we need to have a look briefly DROP1 and DROP3.

In fact, DROP2 is the improvement of DROP1, and DROP3 is applied DROP2 onto the

training set which is already applying ENN.

There are some notations as well as new concepts which we need to consider when

we work with these algorithms as follows:

P_1..n: Training set T consists of n prototypes.

P.N_1..k: the prototype P has k nearest neighbors (ordered from nearest 1 to furthest

k).

P.E: the prototype P has a nearest enemy. The enemy is the nearest instance in the

list of P’s nearest neighbors, but with a different from class labels of P.

P.A_1..a: the list of prototype which have P as one of their kNN, called list of

associate of P.

In terms of DROP1 algorithm, the following basic rules are used to decide

removing an instance from the subset training S is safe or not.

The algorithm will remove prototype P if at least as many of its associates in S

would be classified correctly without P. [reduce techiques]. In order to do this action, each

associate in P.A _1..ais check to see the affection of removing P on it. It is reasonable that

when a prototype P is removed from the subset of training, then the list of associate of each

associates P.A_ineed to be updated. Each associate P.A_iwill use its (k+1)st nearest neighbor

(P.A_i.N_k+1) in place of P.

There are two possible action which can happen after update the list of associates:

the prototype P has the same class as P.A_i, and P.A_i.N_k+1has a different class with P.A_i,

then this situation may lead P.Ai to be misclassified by its neighbors. In other case, the

prototype P has a different class from P.A_iand P.A_i.N_k+1is the same class as P.A_i, then

removing P could cause a previously misclassified instance to be classified correctly.

The pseudocode for DROP1 is given as follows (Shi, Wang, & Weng, 2012):

DROP1:

DROP1(Training set T ): Instance set S.

Let S= T .

For each instance P in S:

Find P:N_1..k+1, the k+1 NNs of P in S.

Add P to each of its neighbors’ lists of associates.

For each instance P in S:

Let with = # of associates of P classified correctly with P as a neighbor.

Let without = # of associates of P classified correctly without P.

If without >= with

Remove P from S.

For each associate A of P

844

Remove P from A’s list of NN

Find a NN for A.

Add A to its new neighbor’s list of associates.

For each neighbor N of P

Remove P from N’s lists of associates.

Endif

Return S.

The potential problem that can arise in DROP1 is that a noisy instance typically has

associates of a different class, so that the noisy instance may occupy the input space.

However, in case of its associates are removed then the noisy instance will keep more input

space. On the other side, if many of its neighbors are removed first then its associates may

contain prototypes, which have the same class, comes from the other side of its original

boundary. From these reasons, it is possible that removing the noisy instance at that point

may lead to some of its associates to be classified incorrectly.

DROP2: using more information and ordering the removal

DROP2 is the improvement of DROP1 by considering the effect of the removal of

a prototype on all the set of prototypes in the original training set T instead of instances

remaining in subset of training S.

The DROP2 algorithm removes prototype P if at least as many of its associates in T

would be classified correctly without P. In order to do that, each prototype P in the

original training set T continues to maintain a list of its k+1 nearest neighbors (NNs) in S

when P is already removed from S. It means that the set of prototypes in S have associates

that are both inside and outside S. This method can solve the problems of removing the

entire clusters, which causes by DROP1.

Furthermore, DROP2 also changes the order of removal of instances by sorting the

prototype S based on the similar distance to their nearest enemy. The instance furthest

from its nearest enemy or furthest from the decision boundary is removed first so that this

removing retains the border points. However, in some cases, noisy instances are also

“border” points, this action can cause the order of removal to be drastically changed.

DROP3: filtering noise

In order to solve problem from DROP2, DROP3 algorithm uses a noise-filtering

pass before sorting the instances in S. This is done using a rule similar to ENN as we

discussed so far. At each step of removing one noisy prototype, the instances in the

training set T are sorted by distance to their nearest enemy remaining in S. It means that

some points far from the real decision boundary are removed first.

We can simply give the pseudocode for DROP3 algorithm as follows:

DROP3

DROP3 = ApplyDROP2(ENN(T))

845

where T is the original training set.

2.3.3 Random mutation hill climbing (RMHC):

This is a local search method which applied genetic algorithm at step of mutation

and getting the highest fitness of population.

The algorithm uses a binary bit string to represent a set of prototypes where the

length of the string is the number of instances in the original training set. The position

where a bit is equal to 1 meaning that at the instance with the index is the position of this

bit in the string. The search technique is that the mutation of the bit vector changes the

selection of instances in the prototype set from the nearest neighbor computation (Skalak,

2010). The fitness function used for all the RMHC experiments is the predictive accuracy

on the training data of a set of prototypes using the 1-NN classification algorithm.

With a given the original training set X, the t_maxiterations, and the number of

instances remains in the subset of training. After the number of iterations t_max, the string

with the highest accuracy is the best subset of training.

The pseudocode for RMHC is given as follows (Skalak, 2010)

RMHC:

Given: The original training set X

Number of iterations t_max

Number of prototypes p

X_best= random selection of p instances from X

b=random bit vector encoding p prototype indices

FOR i=1 to tmax DO

j=random number between 0...|b|

flip but b_j

X’= set of prototype from X included by b

IF kNN(X’,X)>kNN(Xbest,X) THEN

X_best=X’

Return (X_best)

2.2. Methods

In this section we discuss our experimental evaluation.

The research used the email collection as the dataset with 1062 instance and 58052

features for each email. Although the main topic of this study is PS, in order to improve the

speed of PS algorithms, we use the result from feature reductions of some baseline

methods to reduce the number of redundant features; we keep the top 50 features. The

baseline method of feature selection (FS) used in my study is SVM. It means that every

time before PS is carried, FS is ran first to reduce features. This will save time for running

the PS algorithms later.

846

Secondly, we carry out 10-folds crossvalidation and respond the average accuracy

over all folds. the original training set, the test of accuracy the average of ten times of

crossvalidation.

Last but not least, in order to apply 1-NN algorithm, and using the Euclidian

distance function, we cannot apply them directly onto the original dataset because the

original dataset may contain a lot of non-numeric features. That is why at the first step of

the pre-processing procedure, all features have numeric values and Boolean values are

converted to numeric values. After that all feature values are linearly scaled from 0 to 1.

The implementation of this study is divided into two parts as follows:

The first part of implementation: Assessing the dimension reduction and accuracy

of classification of each PS method applied to the original training set. It is ran as follows:

At each iteration, firstly we apply FS, then apply ENN, DROP3, RMHC on the subset of

original instances. Then, we use the kNN to test the accuracy for both original subset of

instances and new subset training of instances after applying PS.

The second part of the implementation instead of using separately three new

subsets of training, the study gets the intersection between each pair of subsets and also the

intersection between three new training subsets in each iteration of the crossvaliadation.

The study also uses kNN to test the accuracy and reduction and compares together as well

as the previous implementation. At the end we can conclude the best intersection which is

the best intersection.

In order to get the intersections of subsets training, in each implementation of each

algorithm, instead of returning directly the subset training, we use a vector to contain

indices of instances which are decided to remain in the subset training. After that in case of

combination, we get the intersection between pair of vectors, and between three vectors.

From the vector of intersections we can easily result the subset training of intersection for

each case. For example, in our study, we use vectors v_ENN, v_DROP3, and v_RMHC.

After that we get the vector intersections of each pairwise vector:

v_ENN_and_v_DROP3,v_DROP3_and__RMHC, v_ENN_and_v_RMHC, and the vector

intersection between three vector subsets training: v_intersection. From these result

vectors, we can obviously get the subsets training for each case. At the last step, we use

these subset training to test the accuracy of classification based on kNN rule, and setting -

up 10-folds crossvalidation.

3. Results and discussion

3.1. Results of the first part of the implementation

847

(a)

(b)

(d)

(c)

Figure 2. (a) Dataset with classes “●” for spam mails and “●” for ham mails

(b) Subset selected by ENN

(c) Subset selected by DROP3 with 90% number of instances

(d) Subset selected by RMHC

The Table 1 shows the result about accuracy, reduction and time cost in both case

of classification with and without PS.

Table 1. The accuracy, reduction and time cost of classification when applying kNN

to original training set, and to the subsets of training after applying ENN, DROP3,

RMHC algorithms

Subset selected by

algorithms

Accuracy

(%)

Reduction (%)

Time

(minutes)

Original training set

ENN

79.46

84.85

84.05

84.51

0

25

2

12.99

54.51

70

DROP3

8

RMHC

9

We can see that in case of classification with whole original training set, the

accuracy of classification based on kNN rule is 79.46% with quite expensive time cost,

848

about 25 minutes because every instance is searched as equal roles. It means that the

classification had to take a long time to check some prototypes which may have no effect

on the classification at all.

More interesting, the accuracy of classification using the kNN rule on the three new

training subsets which result from applying ENN, DROP3 and RMHC to the original

training set are quite similar with 84.85%, 84.05% and 84.51% respectively. In this case, if

we just consider the result of accuracies then it is difficult for us to see the difference

between three PS algorithms. It is better for us to consider the second criteria in the list of

PS’s goal: stored requirements. It is clear that DROP3 has stronger effects on removing

redundant instances than ENN. In order to find the reason for this, we will have a look

again two these algorithms. In fact, ENN removes misclassified instances from the original

training set. Furthermore, DROP3 is the algorithm applying DROP2 on the training subset

which is already filtered. The removing in DROP3 is done in a careful way. It prevents

both the central points as well as the “smooth” border points of the clusters based on the

order in the list of associates and nearest neighbors. It results that the important instances

are remained in the subset of training, and almost redundant instances are removed from

the training set as much as possible. That is why in our training set, by applying DROP3,

54.51% of number instances are removed while the accuracy does not change much

comparing with the case of applying ENN.

In case of RMHC, as we know, the direction of search is fixed. It means that the

users can fix the number of instances in their new training sets. The subset with the fixed

number of instances with highest fitness is chosen as a new training subset. In fact, the

number of instances which we fixed so far has an effect on the accuracy of the

classification later. For example, if we remove too many instances of the original training

set, then the new training set sometimes does not have enough necessary characteristics,

it means that we will not have much choice to consider which instances are much better

than others, while if we keep too much instances in the subset of training then a number

of instances are redundant, they cause waste time for classification. For instance, the

results of accuracy in case of reduce 90%, 50%, 30% and 10% of the are much different

with 92.87%, 82.41%, 84.51% and 78.37% respectively. In our study, we apply RMHC

with 30% instances, and its accuracy of classification is approximately the average of the

other cases.

In case of the time requirements, the DROP3 takes much longer that ENN.

However, in the case of RMHC, it depends on the number of iterations as well as

remaining instances numbers.

3.2. Results of the second part of the implementation

849

Figure 3. Subset selected by getting the intersection of three new training sets, which

obtained from ENN, DROP3, and RMHC

The study implemented the intersection between each pairwise of training subsets

and between three training subsets, then testing the accuracy. The results getting from the

implementation as following:

Table 2. The accuracy, reduction and time cost of the classification for intersections

Intersection

Accuracy (%)

Reduction (%)

Time (minutes)

ENN-RMHC

83.68

69,73

11

DROP3-RMHC

ENN-DROP3

82.84

95.42

92.31

61.85

59.67

87.78

17

10

19

ENN-RMHC-DROP3

Consider the results of intersecting training subsets, we can see that the case of

intersection between ENN and DROP3 results in the best accuracy (95.42%) with

reasonable time requirement and a large reduction, nearly 60%.

Compare with the Table 1 we can see that the intersections have more accuracy

applying to separated algorithm. Especially, the intersection between ENN and DROP3 got

a much higher accuracy than the highest accuracy in case of applying individual algorithms

(84.85%) while the time increase is not much. Because ENN takes a short time to

implement, the time cost depends mostly on DROP3.

4. Conclusion

We have presented analyses of three algorithms: ENN, DROP3, and RMHC, and

more importantly we have combined them into one method by getting the intersection of

new subsets of training. The results show that the accuracy of classification on our

intersection subsets training are much higher than the accuracy of classification on the

subset training which results from separated algorithms. The best accuracy of classification

on the subset training which results from intersecting ENN and DROP3 subsets can be

assessed as a good method of PS. From this study, it may open a new direction of studies

in the PS problems. Instead of trying work with separated algorithms, we also should try to

find the best combination between some of them.

850

TÀI LIỆU THAM KHẢO

1. Aggarwal, M., & Amrita. (2013). Performance Analysis Of Different Feature Selection

Methods In Intrusion Detection. International journal of scientific & technology

research.

2. Alejo, R., Sotoca, J., & Valdovinos, R. (2010). Edited Nearest Neighbor Rule for

Improving Neural Networks Classifications. In Advances in Neural Newtwork - ISSN

2010.

3. Bien, J., & Tibshirani, R. (2012). Prototype selection for interpretable classification.

The Anuals of Applied Statistics.

4. Bo, T., & Haibo, H. (2015). ENN: Extended Nearest Neighbor Method for Pattern

Recognition. Computational Intelligence, 52-60.

5. John , C., & Eliot, M. (2006). Hybrid Optimizations: Which Optimization Algorithm

to Use? Compiler Construction, (pp. 124-138). Vienna.

6. Lu, S., Jie, Y., Wuyang, W., & Shasha, L. (2017). Instance selection by random

mutation hill climbing algorithm with MapReduce in big data. Computer Science, 252-

259.

7. Nele, V., & Cornelis, C. (2013). A Fuzzy Rough Prototype Selection method.

Information System.

8. Shi, L., Wang, Q., & Weng, M. (2012). Spam Email Classification Using Decision

Tree Ensemble.

9. Skalak, D. (2010). Prototype and Feature selection by sampling and random mutation

Hill Climbing Algorithms.

10. Wilson, D., & Martinez, T. (2000). Reduction techniques for instance-Based learning

Algorithms. Neural Network & Machine Learning, (pp. 257-286).

851