3.2 illustrates an effect of the size of the training set on the possible difference between posterior and expected false discovery rates. Black dots on both plots show expected versus posterior FDR for randomly sampled training and test sets. Continuous lines represent regression of the mean expected vs. the mean posterior FDR. Both regression lines are averaged over 1000 randomly sampled sets. On average, expected and posterior false discovery rates are about the same for both sizes of the training set. On the other hand, random differences between expected and posterior FDR are much larger for the training sets with a smaller number of records.
Table 3.1 shows mean values of posterior false and true discovery rates of three AD methods, applied for detection of the gas-filled sand anomaly. To get mean values shown in the Table 3.1, 1000 randomly sampled training and test sets were created and each pair of the training-test sets was used for calculation of posterior true and false discovery rates. An expected false discovery rate was assigned the value of 20%. According to the Table 3.1, the mean value of the posterior false discovery rate is about the same and very close to the expected false discovery rate for all three AD methods. On the other hand, posterior true discovery rate is different using different methods and is the largest for the divergence classifier.
Figure 3.2 Expected versus posterior false discovery rate for two sizes of the training set. Classifier: distance from the center of the training set. Coordinates of a record: Poisson’s Ratio and Vp/Vs.
Table 3.1 Mean values of false and true discovery rates for three classifiers in detection of gas-filled sands anomaly. Size of the training set: 20 records.
| AD method | Expected false discovery rate (%) | Mean of posterior false discovery rate (%) | Mean of posterior true discovery rate (%) |
|---|---|---|---|
| Sparsity | 20 | 20.44 | 60.85 |
| Distance | 20 | 20.44 | 71.32 |
| Divergence | 20 | 20.49 | 85.71 |
Since the divergence classifier was specifically designed for detection of gas-filled sand anomalies, it outperforms two universal classifiers. Its mean the posterior true discovery rate is as high as 85%.
3.5 ROC Curve Analysis
The ROC curve analysis presented in this section is based on the analysis of results of anomaly detection with 1000 bootstrap sampled training and test sets of different size. Two input petrophysical parameters, Poisson’s Ratio and Vp/Vs, were used for calculation of AD classifiers for each of the pair training - test sets. For every training set, the expected value of the false discovery rate was assigned, and distributions of posterior true and false discovery rates were calculated on anomaly detection results.
In the case of multiple bootstrap sampled training and test sets, ROC curve analysis has to deal with multiple ROC curves. In that case, quantiles of the true and false discovery rates, and quanties of area under ROC curve may be used for a compact description of a set of multiple ROC curves.
The writers distinguish two types of ROC curve analysis: (a) posterior ROC curve analysis, which is identical to traditional ROC curve analysis and (b) expected - posterior ROC curve analysis. Posterior ROC curve is a plot of posterior true discovery rate versus posterior false discovery rate, both calculated on the test set. Expected-posterior ROC curve is a plot of posterior true discovery rate (posterior TDR) versus expected false discovery rate (expected FDR), where posterior TDR is calculated on the test data and expected FDR assigned using training set data.
Figure 3.3 shows histograms of posterior AUC values for three classifiers - divergence, sparsity, and distance. Sets of values of AD classifiers were created with bootstrap random sampling of the training and test sets. Each training set contained five records. The total number of randomly generated pairs of training and test sets was 1000. The intersections of continuous vertical black lines with x-axis at Figure 3.3 shows the median value of the AUC. Dashed lines indicate lower and upper quantiles of the AUC distribution calculated for quantile probabilities Plow = 0.1 and Pupper = 0.9. According to the Figure 3.3, the divergence classifier is characterized by the narrowest distribution of AUC values and the largest AUC median. Distribution of this classifier is symmetric. Distributions of distance and sparsity classifiers are asymmetric with long tails in the direction of smaller values. As a result, lower quantiles for the distribution of distance and sparsity AUC are shifted towards smaller AUC values.
Figure 3.3 Histograms of area under posterior ROC Curve (AUC) for three anomaly detection classifiers. Size of each training set is 5 records.
Two of the three AD classifiers with AUC shown in Figure 3.3 do not rely on the use of information about properties of petrophysical parameters within potential anomalies. This is their advantage since they may be used for detection of any type of anomaly with unknown properties. Although they underperform compared to the divergence classifiers that rely on the use of known anomaly properties, they still can produce median AUC values as high as 0.75.
Figure 3.4 shows posterior AUC histograms for sparsity classifiers calculated for two sizes of the training sets. Continuous vertical lines are median values, intersection of doted lines with x-axis are the levels Plow=0.1 and Pupper=0.9. According to Figure 3.4, the AUC quantile region in the case of 25 records training sets is 40% narrower than the quantile region in the case of the training sets of 5 records.
Figure 3.4 AUC histograms and quantile regions calculated from 1000 pairs of training-test sets. Size of the training sets: 5 and 25 records. Sparsity classifier.