3.5 Median of posterior AUC for three AD classifiers as a function of the size of the training set.
The median of posterior AUC values and the width of the AUC quantile region for three AD classifiers are shown in Figures 3.5 and 3.6. According to Figure 3.5, the median AUC of the divergence is higher than median AUC of other classifiers with its values stable around 0.9. Importantly, it is as high as 0.9 even for small training sets containing only five records. The sparcity classifier has the lowest AUC median which may be as small as 0.70.
Figure 3.6 Quantile width of AUC distribution calculated on anomaly detection results from 1000 randomly generated pairs of training-test sets.
According to Figure 3.6, divergence has the narrowest AUC quantile region. AUC quantile regions of two other AD classifiers are significantly wider than that of divergence. The width of AUC quantile regions for these classifiers decreases with the increasing size of the training sets. It is still about three times as wide as the AUC quantile width of divergence for the size of training set 20, 25 records.
3.6 Optimization of Aggregated AD Classifier Using Part of the Anomaly Identified by Universal Classifiers
It was shown in the previous section that a divergence AD classifier is very efficient in detecting an anomaly with known properties. It was noted also that universal classifiers have lower efficiency compared to divergence.
The goal of this section is to develop a methodology for construction of an adaptive AD classifier working as a universal classifier for detection of anomalies of unknown type that is still almost as efficient as the specialized divergence classifier. The methodology for its synthesis is a two-step procedure:
1 1. Detection of a part of the anomaly using universal classifiers, such as the distance or the sparsity.
2 2. Optimization of aggregated classifier on the detected part of anomaly.
The structure of the aggregated classifier is defined by a set of coefficients sm (Eq. 3.7). These coefficients are chosen so that they maximize the ratio:
where sm; 1 ≤ m ≤ M are weights and aggr() is defined by Eq. 3.7, classifier Anomaly Records are records identified by a universal classifier as anomaly, trainSetRecords are records from the training set.
To find coefficients sm that maximize efficiency criterion (3.8) we used a multidimensional grid search. In the grid search, the coefficients sm in Eq. 3.7 take on a discrete set of values in the following region:
(3.9)
The efficiency criterion (3.8) is calculated for each combination of coefficients sm at individual grid nodes. An adaptive aggregated AD classifier maximizes efficiency criterion on the grid. As soon as the aggregated classifier is synthesized it may be used for anomaly detection using the test set.
We illustrate construction of an adaptive aggregated classifier using the sparsity classifier at the first optimization step. The classifier to be synthesized is of the following form:
(3.10)
Therefore, search is done on the two-dimensional grid.
Figure 3.7 shows values of the sparsity classifier used as the first step of optimization of the aggregated classifier. Twenty regular records that form the training set are randomly selected out of a set of 50 records. The test set contains 30 regular and 25 anomaly records. Anomaly records are from gas-filled sands. Regular ones are from brine-filled sand or shale. The assigned value of the expected false discovery rate was 20%. Records classified as a potential anomaly include 13 actual anomaly records and 2 regular records. Thus the posterior true discovery rate is very moderate - 52%.
Figure 3.7 Sparsity values on the records of the training and test sets. Horizontal dashed line - anomaly detection cutoff producing an expected false discovery rate of 20%.
Figure 3.8 Anomaly detection. Histograms of posterior true discovery rate (TDR) for two values of expected false discovery rate. AD method: aggregated. 20 regular records in each training set.
Table 3.2 Mean and three quantiles of distribution of AUC values.
| Mean | Quantiles | Width of quantile region | |||
|---|---|---|---|---|---|
| P=0.05 | Median P=0.5 | P=0.95 | |||
| Divergence | 0.897 | 0.89 | 0.895 | 0.909 | 0.019 |
| Aggregated | 0.866 | 0.862 | 0.87 | 0.901 | 0.039 |
| Distance | 0.795 | 0.633 | 0.818 | 0.885 | 0.252 |
| Sparsity | 0.765 | 0.576 | 0.786 | 0.868 |
0.292
|