GeinQSAR Studio
Home › The classifier dashboard

The classifier dashboard

The report of a classification model: the statistics of every set, the label-permutation test, the ROC curve and confusion matrix, every compound and the applicability domain.

The classifier dashboard opens after a classification workflow. Like the model dashboard it computes nothing of its own: the cards, the tables and the figures are read from the metric table and the outputs of the validation node, so they agree with the exported report and the files of the run. The class labels are those of the class column, or active and inactive when the classes were derived from a threshold.

It has four tabs: Overview, Curves, Compounds and Domain, with Export report at the right of the tab bar. The inspector summarises the classifier: model, compounds per set, number of classes, cross-validated accuracy and MCC, the number of compounds outside the domain and the descriptors.

Overview

Performance is one card per statistic with one line per set: Training, Cross-validation, Test and, when there is one, External. Read the cross-validation and test lines. A random forest reproduces its own training compounds almost perfectly, so its training line is close to 1.000 everywhere and says nothing about prediction.

The Overview of a random forest on 100 compounds (75 training, 25 test). Every training value is 1.000, which says only that the forest reproduces the compounds it was grown on. Cross-validation gives an accuracy of 0.667 but an MCC of only 0.122 and a sensitivity of 0.160 against a specificity of 0.920: the model assigns nearly every compound to the larger class. The test line looks far better, with an MCC of 0.915, but it rests on 25 compounds. A difference this large between the two estimates means that at least one of them is not to be trusted.
The Overview of a random forest on 100 compounds (75 training, 25 test). Every training value is 1.000, which says only that the forest reproduces the compounds it was grown on. Cross-validation gives an accuracy of 0.667 but an MCC of only 0.122 and a sensitivity of 0.160 against a specificity of 0.920: the model assigns nearly every compound to the larger class. The test line looks far better, with an MCC of 0.915, but it rests on 25 compounds. A difference this large between the two estimates means that at least one of them is not to be trusted.
StatisticDefinitionReference
Accuracy(TP + TN) / n: the fraction of compounds assigned to their observed class. Inflated by a majority class.
Balanced accuracyMean of the recall of every class. A majority class cannot inflate it; 0.5 is chance for two classes.Brodersen et al., Proc. ICPR 2010, p. 3121
MCCMatthews correlation coefficient: (TP TN - FP FN) / sqrt((TP + FP)(TP + FN)(TN + FP)(TN + FN)). +1 is a perfect prediction, 0 is chance, -1 the inverse. The single best summary for unequal classes.Matthews, Biochim. Biophys. Acta 405:442 (1975); more than two classes: Gorodkin, Comput. Biol. Chem. 28:367 (2004)
Cohen kappaAgreement of observed and predicted classes beyond what chance alone would give: 1 perfect, 0 chance.Cohen, Educ. Psychol. Meas. 20:37 (1960)
SensitivityTP / (TP + FN): the true positive rate, or recall, of the positive class.
SpecificityTN / (TN + FP): the true negative rate.
PrecisionTP / (TP + FP): the positive predictive value.
F1Harmonic mean of precision and sensitivity.van Rijsbergen, Information Retrieval, 2nd ed., Butterworths 1979
ROC AUCArea under the receiver operating characteristic curve: the probability that a positive compound scores above a negative one. 0.5 is chance. Independent of any threshold on the probability.Hanley and McNeil, Radiology 143:29 (1982)
ROC AUC lower 95 %, ROC AUC upper 95 %The limits of the confidence interval of the AUC, from the DeLong standard error.DeLong, DeLong and Clarke-Pearson, Biometrics 44:837 (1988)
Log lossMean of -ln(probability given to the observed class). 0 is a perfect, confident prediction; ln 2 = 0.693 is what a two-class model scores when it always answers 0.5. A confident wrong probability is punished hardest.Good, J. R. Stat. Soc. B 14:107 (1952)

TP, FP, TN and FN count true and false positives and negatives. For two classes, sensitivity, specificity, precision and F1 refer to the positive class named in the Model section, which is a parameter of the validation node (see Classification models). With more than two classes the cards show the unweighted (macro) means of precision and F1 instead, and the Per class table gives sensitivity, specificity, precision and F1 of every class against the rest, for every set. A statistic the run could not define is left empty: a set with one class has no MCC and no ROC AUC, and a model without class probabilities has no log loss. The warnings of the run say which statistics were undefined and why.

Below the cards, Model lists the model and how it was fitted, the response, the classes, the positive class, the compounds per set, the number of descriptors and the cross-validation scheme with its number of resamples. Then come Per class, Label permutation, the warnings and All metrics, the complete metric table with Save as CSV.

The Overview of the BBBP classifier (1474 training, 491 test compounds, 3976 descriptors, three repeats of stratified 5-fold cross-validation, 15 resamples). Cross-validated and test values agree closely (balanced accuracy 0.800 and 0.803, MCC 0.684 and 0.689, ROC AUC 0.931 and 0.943), the sign of a model that predicts as well as it validates. The label permutation section shows an observed MCC of 0.6837 against a mean of -0.0049 over 25 shuffles.
The Overview of the BBBP classifier (1474 training, 491 test compounds, 3976 descriptors, three repeats of stratified 5-fold cross-validation, 15 resamples). Cross-validated and test values agree closely (balanced accuracy 0.800 and 0.803, MCC 0.684 and 0.689, ROC AUC 0.931 and 0.943), the sign of a model that predicts as well as it validates. The label permutation section shows an observed MCC of 0.6837 against a mean of -0.0049 over 25 shuffles.

Label permutation

The cross-validated MCC with the true labels is compared with the same score after shuffling the labels. The section gives the number of permutations and of failed ones, the MCC with the observed labels, the mean and the largest MCC with permuted labels, and the p value: (1 + number of shuffles scoring at least as well) / (shuffles + 1) (Ojala and Garriga, J. Mach. Learn. Res. 11:1833, 2010). It passes below 0.05. The table lists the true labels in its first row and every shuffle after it.

The label-permutation section of the 100-compound model. The cross-validated MCC with the true labels, 0.1222, is beaten by one of the shuffles (0.1543), and the p value of 0.3333 fails: this model cannot be told apart from chance, whatever its test line says.
The label-permutation section of the 100-compound model. The cross-validated MCC with the true labels, 0.1222, is beaten by one of the shuffles (0.1543), and the p value of 0.3333 fails: this model cannot be told apart from chance, whatever its test line says.

Curves

The Curves tab shows the ROC curve and the confusion matrix of one set. The Set drop-down chooses it and starts on Cross-validation, the set every training compound is predicted in without having been fitted on. The subtitle of the ROC curve gives the AUC with its 95 % confidence interval. The confusion matrix has the observed class in rows and the predicted class in columns, so the diagonal holds the correct predictions. The camera button of each chart opens the figure dialog.

The ROC curve of the 100-compound model in cross-validation: AUC 0.754 with a 95 % confidence interval from 0.643 to 0.865. The curve lies above the dashed diagonal of a random ranking, but the interval is wide for 75 compounds.
The ROC curve of the 100-compound model in cross-validation: AUC 0.754 with a 95 % confidence interval from 0.643 to 0.865. The curve lies above the dashed diagonal of a random ranking, but the interval is wide for 75 compounds.
The cross-validated ROC curve of the BBBP classifier as a print figure: AUC 0.931 (0.915 to 0.946). The curve rises steeply, so a high threshold on the probability gives few false positives.
The cross-validated ROC curve of the BBBP classifier as a print figure: AUC 0.931 (0.915 to 0.946). The curve rises steeply, so a high threshold on the probability gives few false positives.
The confusion matrix of the BBBP test set. Of 375 compounds of class 1, 366 are recognised; of 116 compounds of class 0, 42 are predicted as class 1. The errors fall mostly on the smaller class.
The confusion matrix of the BBBP test set. Of 375 compounds of class 1, 366 are recognised; of 116 compounds of class 0, 42 are predicted as class 1. The errors fall mostly on the smaller class.

Compounds

One row per compound: set, observed and predicted class by their labels, whether the prediction is Correct, one probability column per class, P(label), the cross-validated prediction with its probabilities and how often the compound was held out, and the domain flag. Save CV predictions writes the cross-validated predictions. The table shares its selection with the charts, as in the model dashboard.

The Compounds tab. For the training rows shown every prediction is correct, with P(blocker) from 0.017 to 0.867; the cross-validated columns further right are the ones that show where the model fails.
The Compounds tab. For the training rows shown every prediction is correct, with P(blocker) from 0.017 to 0.867; the cross-validated columns further right are the ones that show where the model fails.

Domain

The validation report of a classifier has no domain assessment of its own. The dashboard asks the trained model for the domain verdict of every compound, so the verdicts are the ones a later prediction with this model would give. The tab shows the compounds inside per set, one card per method with the share of each set that method alone places outside, and the per-compound table with Outside only and Save as CSV. When the model was trained without an applicability domain the tab says so.

The Domain tab: 74 of 75 training and all 25 test compounds inside. The classifier recipe has no leverage, so the only method card is the kNN distance, which places 9.3 % of the training set outside.
The Domain tab: 74 of 75 training and all 25 test compounds inside. The classifier recipe has no leverage, so the only method card is the kNN distance, which places 9.3 % of the training set outside.

Another example

Another run of a hERG classifier with 75 training and 25 test compounds. The pattern is the same: perfect training values, a cross-validated MCC of 0.065 and a test MCC of 0.831. Small sets give unstable test statistics; trust the cross-validation and the permutation test.
Another run of a hERG classifier with 75 training and 25 test compounds. The pattern is the same: perfect training values, a cross-validated MCC of 0.065 and a test MCC of 0.831. Small sets give unstable test statistics; trust the cross-validation and the permutation test.
An exported report of a classifier carries the same cards, the per-class table, the permutation scores and the figures. See Reports and export.