GeinQSAR Studio
Home › Accuracy and validation

Accuracy and validation

What was measured to establish that GeinQSAR Studio computes what it claims, the results, the conditions under which they hold, and where they fall short.

The validation covered four aspects: the reading of real structures by the chemistry layer, the agreement of the learning algorithms and the printed statistics with independent implementations, the accuracy of the models relative to established tools on the same data, and the comparison of the 3D-QSAR methods on published benchmark sets. Where a comparison needed an independent program, an established reference implementation was used: an open cheminformatics toolkit for structures and descriptors, and a machine-learning library for the learning algorithms. Neither is part of GeinQSAR Studio. All results on this page were obtained with the program itself.

The public datasets were ESOL, FreeSolv, the Lipophilicity set of MoleculeNet, BACE, BBBP, an Ames mutagenicity set, acetylcholinesterase and hERG sets curated from ChEMBL with the program, and the eight sets of Sutherland, O'Brien and Weaver (J. Med. Chem. 2004, 47, 5541) with their alignments and their division into training and test compounds.

Two definitions matter when these numbers are compared with published 3D-QSAR results. The cross-validated q2 is computed with the sums over the training compounds. The predictive r2 of a test set, identical to Q2F1, is computed with the sums over the test compounds and the mean of the training set.

Structure handling and stereochemistry

The 17 057 unique SMILES of the public sets were read by the program and by the reference implementation. The program parsed all of them, the reference 17 046; the remaining 11 entries of BBBP have valences that the reference rejects. For the molecules parsed by both:

QuantityComparedAgree%
SMILES parsed by GeinQSAR Studio17 05717 057100.00
SMILES parsed by the reference17 05717 04699.94
Heavy-atom count17 04617 046100.00
Net formal charge17 04617 046100.00
Ring count (SSSR)17 04617 046100.00
Aromatic atom count17 04617 02199.85
Assigned tetrahedral centres17 04617 04599.99
Assigned stereo double bonds17 04617 046100.00
Unassigned tetrahedral centres17 04617 01399.81
Unassigned stereo double bonds17 04616 85198.86
CIP descriptors of atoms (R/S, r/s)7 6227 622100.00
CIP descriptors of double bonds (E/Z)1 0191 019100.00
Canonical SMILES re-read: same InChIKey17 04617 04599.99
Canonical SMILES invariant to atom order17 04617 04599.99
SD reader: stereo-free InChIKey equal (Sutherland files)1 3371 337100.00

Canonical SMILES written by the program and read back by the reference gave the InChIKey of the input for all molecules but one, and they did not change when the atoms of each molecule were reordered at random five times, again with one exception. Aromaticity differed in 0.15 % of the molecules, which reflects the two aromaticity models. Six defects of the chemistry layer were found by this comparison and corrected before the final run.

Descriptor concordance

Descriptors with the same published definition in both programs were compared on the same 17 046 molecules. The table gives the percentage of molecules with identical values (difference at most 10-6), as the median and the minimum over the descriptors of each family.

FamilyDescriptorsMedian %Minimum %Main cause of differences
Constitutional counts and masses5100.055.4atomic-weight tables; electron mass of ions
Ring perception10100.099.5definition of bridgehead atoms in cage systems
Aromaticity-dependent counts399.599.5aromaticity model
Hydrogen-bond counts4100.061.9pattern definitions (S-H donors; acceptor definition)
Topological polar surface area292.092.0nitro group entry of the table (2.68 Å2 per group)
Wildman-Crippen log P and MR299.099.0P=S sulfur type; aromaticity model
Connectivity indices, simple2100.0100.0none
Connectivity indices, valence593.486.6formal charge in delta-v; ring subgraphs counted as paths by the reference
E-state indices393.593.2formal charge in delta-v
Distance-based indices5100.0100.0none
Rule counts299.999.2threshold cases of the ingredients
Rotatable bonds295.382.7definition (ester C-O, nitro, bonds next to triple bonds)
Kappa shape indices, Hall-Kier alpha424.910.3hybrid state of conjugated N/O; kappa3 formula for even atom counts
QED327.326.6structural alerts, acceptor and rotatable-bond definitions
Balaban J (the reference's variant)14.54.5the reference weights distances by bond order
Bertz complexity10.00.0different conventions; no published numeric table
Labute ASA10.00.0radius set
Gasteiger charges (atoms within 0.001 e)156.656.6hybrid state assigned to conjugated N and O
MACCS keys (agreement per key)166100.054.6reconstructed key definitions differ for ring and CH2 patterns; aromaticity
ECFP4 (number of environments)199.799.7hash values differ by design; the same number of atom environments is compared

Atom, ring and Lipinski counts, the simple connectivity indices and the distance-based indices are identical. TPSA and the Wildman-Crippen values agree for 92 % and 99 % of the molecules; the differences were traced to the entry used for nitro groups and to the atom type of thiophosphoryl sulfur. Valence connectivity and E-state indices differ only for molecules with charged atoms, where the two programs treat the formal charge differently, and are identical on the subset without charged atoms.

Where the definitions themselves differ (rotatable bonds, pattern-based acceptors, the Hall-Kier alpha, Labute surface area, Bertz complexity) the values correlate but are not equal; these are documented conventions, not errors. QED is a case in point: the program applies the published formula of Bickerton (2012) to its own structural alerts and acceptor definition, so its values correlate with those of the reference (r = 0.92) without reproducing them. Compare QED values only within GeinQSAR Studio.

Dataset curation

CheckResult
p-scale value against the pChEMBL value published by ChEMBL41 193 of 41 193 records agree (62 628 activity records for three targets downloaded; 41 193 carry a pChEMBL value)
Standardised structure against the largest-fragment and neutralisation procedure of the reference, by InChIKey37 303 of 37 305 identical, including all 1 976 salts and solvates; the other 2 are sulfoximines that the program normalises by design
Datasets regenerated from their provenance files13 of 13 with the recorded checksum

As an example of what the curation removes: for human acetylcholinesterase (ChEMBL release 37), 10 079 IC50 records were reduced to 5 627 compounds. 892 records were censored, 1 361 had no convertible value, 1 370 were flagged by the database as invalid or as repeated citations, and the rest were merged as replicates.

The first use of the Dataset Builder on live data exposed a defect, corrected before release: PubChem data tables that cover several proteins had been pooled under the first target, censored values were read as exact, and the activity type of each row was ignored. Records now carry the target, relation and type of their own row, and a multi-target table is curated only after one target has been chosen.

Learning algorithms and reported statistics

CheckResult
Each learning algorithm against its counterpart in the reference library, on identical matrices, folds and hyper-parameters, with tolerances fixed beforehand46 of 54 comparisons within tolerance, most of them to rounding error
Statistics printed in the validation reports (training, cross-validation, test, response randomisation and domain statistics, regression and classification), recomputed from the exported predictions with independent code13 355 of 13 355 agree within 10-9 (largest difference 5.2 x 10-13)
Robustness scenarios through the command line: input formats, missing and censored activities, bad values, set sizes from one compound upwards, every model, split and cross-validation scheme205 of 205 behave as specified

The 46 agreeing comparisons include multiple linear regression, ridge, PLS, principal component regression, Gaussian process and support vector regression, logistic regression, discriminant analysis and the tree ensembles. The other 8 have an established cause: the order of tied neighbours, the default convergence tolerance of the coordinate-descent solvers, and variance floors on constant columns. The comparison found one defect, a near-constant column that the internal standardisation of ridge regression divided by rounding noise; it was corrected. The robustness scenarios pass in all 205 cases, after twelve defects found by their first pass had been corrected. A thirteenth was found late: on some charts of a stepwise multiple-regression model, and of a PLS model asked for far more components than the data support, chart labels overlapped, and the report then refused to write itself. The chart now leaves out a label it cannot place clear of its neighbours, rather than drawing it over them, and those reports are written.

2D benchmarks on public datasets

The default recipes of the 2D-QSAR wizard were chosen by repeated cross-validation, with every fitted step inside the folds, on twelve datasets: acetylcholinesterase, ESOL, FreeSolv, hERG and the eight Sutherland sets. On the eleven regression sets among them the default recipe reached a mean Q2 of 0.61, the interpretable variant 0.55, and a recipe that selects six descriptors from all columns before a three-component PLS model 0.38.

Lipophilicity, BACE, BBBP and the Ames set took no part in that choice, so the accuracy on them is independent of it. The benchmark comprised 1 084 runs without a failure. The table gives the test-set accuracy on identical training and test rows for the default recipe, for the best fixed recipe of the program, and for the best recipe of the reference implementation (its descriptors with its models). Random splits: mean ± standard deviation over five stratified 80/20 splits; scaffold splits: one split.

DatasetTraining / testStatisticDefault recipeBest fixed recipeReferenceLiterature range
Not used in the choice of recipes
Lipophilicity3360 / 840RMSE0.713 ± 0.0380.669 ± 0.027 (svr)0.642 ± 0.022 (svr)0.66 to 0.90
BACE pIC501211 / 302RMSE0.701 ± 0.0250.685 ± 0.027 (svr)0.714 ± 0.039 (gbm)
BACE classes, scaffold split1210 / 152AUC0.8320.839 (knn)0.831 (knn)0.78 to 0.87
BBBP, scaffold split1631 / 204AUC0.7650.731 (svc)0.708 (logreg)0.67 to 0.73
BBBP1631 / 408AUC0.938 ± 0.0100.936 ± 0.011 (rf)0.943 ± 0.007 (rf)
Ames mutagenicity5822 / 1456AUC0.916 ± 0.0050.914 ± 0.006 (rf)0.914 ± 0.005 (rf)
Used in the choice of recipes
ESOL903 / 225RMSE0.675 ± 0.0510.552 ± 0.048 (gbm)0.564 ± 0.060 (gbm)0.58 to 1.53
FreeSolv514 / 128RMSE1.162 ± 0.1221.043 ± 0.061 (gbm)0.974 ± 0.105 (gbm)1.15 to 2.11
Acetylcholinesterase pIC50343 / 86RMSE0.776 ± 0.0520.767 ± 0.065 (svr)0.822 ± 0.040 (gbm)
hERG classes443 / 111AUC0.871 ± 0.0300.843 ± 0.029 (gbm)0.819 ± 0.031 (svc)

Recipe names: svr, support vector regression; gbm, gradient boosting; rf, random forest; knn, k-nearest neighbours; svc, support vector classifier; logreg, logistic regression. The literature range is the range of the test-set values of the models reported with MoleculeNet, obtained with another protocol (an 80/10/10 split with hyper-parameter tuning); it is given for orientation only.

On identical rows the program and the reference were equally accurate: neither was better on every set, and the differences were within the variation between splits. The linear default recipe was less accurate than the best non-linear recipe on the three physicochemical sets, by 0.04 to 0.12 in RMSE; that is the price of a model whose latent variables can be inspected.

Applicability domain

With the default recipes, 88 % of the test compounds (mean coverage 0.876) lay inside the applicability domain on average over 17 datasets, and the compounds outside it had the larger error in 11 of them. When models for one target were given compounds of other targets and of the physicochemical sets, the Tanimoto-based domain of the default recipe placed 99 % of them outside.

Conditions to state with any use of these numbers. The recipes were chosen by cross-validation on acetylcholinesterase, ESOL, FreeSolv, hERG and the eight Sutherland sets, so only Lipophilicity, BACE, BBBP and Ames are independent of that choice. The domain rule was chosen by its coverage on all the sets, so its coverage is not an independent result anywhere. The best fixed recipe was selected on the test statistic in both arms, which favours both equally. For the acetylcholinesterase and hERG sets, which pool measurements from many laboratories (the shipped acetylcholinesterase sample pools IC50 values from 43 publications), even the best models stay below the customary thresholds for external predictivity; the validation report says so rather than relaxing a criterion.

How precisely a figure of a small series can be read

The public sets above hold hundreds or thousands of compounds. A congeneric series of a medicinal chemistry project holds tens, and that changes what its statistics mean rather than only how large they are. The question was put to 90 independent ChEMBL series of 32 to 131 compounds, 40 of them chosen and frozen with their checksums before any model was run on them.

Two properties of a data set went with poor prediction, and they can be read off the data before a model is built. Where the activity spread was narrow, a standard deviation below about 0.75 log units, the models beat the training mean on 12 of 20 such series with a median Q2F1 of 0.19. Where activity cliffs were dense, more than about 0.5 cliff pairs per compound, they did so on 3 of 6. Series size made no difference in the 32 to 131 range. These two observations are exploratory: they were found while looking for a cause and were not part of a pre-registered comparison, so treat them as a warning to read the data first, not as a rule. A narrow activity range limiting what a model can achieve is established in the literature as well (Tropsha, Mol. Inf. 2010, 29, 476).

The program applies these two observations itself. The Data page of the 2D-QSAR wizard measures the activity standard deviation and the cliff density of a series as soon as its activity column is chosen, and shows an amber note when either crosses its threshold; the note never blocks a model, and its tooltip gives the figures above together with the measured values of your series. The report carries the same note as its Series guidance row. See The 2D-QSAR wizard, and the cliff analysis for a closer look at the cliffs themselves.

Six model families known to suit small sets were compared against the default on these series, in a comparison whose rule was fixed in advance: the default PLS recipe, PLS without hashed fingerprints, a Gaussian process on a MinMax kernel over count fingerprints, a random forest, support vector regression and the training mean as a reference arm. On the series already seen, three of them looked better than the default. On 40 fresh series, frozen with their checksums before any candidate was run on them, none held up, and the default was left unchanged. That gap between the two halves is the selection effect a fresh confirmation set exists to catch, and it is the reason the defaults of this program are not chosen on the data they are reported on.

3D-QSAR: the benchmark of 59 data sets

KEFRA and the atom-based cubes were compared on 59 data sets, with the comparison and its test fixed before KEFRA was run on them. The two performed alike: the median test Q2F3 was 0.28 for KEFRA and 0.21 for the cubes, a difference that was not significant. KEFRA ran a whole pass of the benchmark in 23 s, where the cubes took 69 minutes, and it reports a prediction uncertainty and an applicability domain for every compound.

On the eight datasets of Sutherland, O'Brien and Weaver, modelled with the authors' alignments, partial charges, lattice boxes and training and test sets, the atom-based model reached, with its own rule for the number of components, a mean q2 of 0.59 and a mean predictive r2 of 0.50.

Sensitivity to choices that are seldom reported

Shifting the lattice by fractions of its spacing changed the q2 of a lattice field model by 0.02 to 0.13 within a set and the predictive r2 by 0.03 to 0.26; rotating the aligned set changed q2 by up to 0.32, in line with the dependence on orientation described by Cho and Tropsha. Differences of this size between two implementations of a lattice method are therefore within what the placement of the lattice alone can produce. When comparing a model of your own with a published one, report the lattice placement, the orientation and the reference molecule of the alignment.

3D-QSAR: the steroid benchmark

The corticosteroid-binding globulin steroids (31 compounds) are the classical 3D-QSAR steroid benchmark of Cramer, Patterson and Bunce (1988). The structures were built from SMILES by the program and verified against the corrected data of Coats. The maximum common substructure found by the alignment is the complete steroid nucleus with the oxygen at position 3, and with conformer selection the nuclei of the nineteen 4-en-3-one steroids superpose with a mean deviation of 0.08 Å.

An aligned steroid series as the program draws it, with the reference molecule in balls and sticks and the others as thin lines. Look at the ring atoms: the nuclei lie on top of one another, while the substituents at the two ends take different directions.
An aligned steroid series as the program draws it, with the reference molecule in balls and sticks and the others as thin lines. Look at the ring atoms: the nuclei lie on top of one another, while the substituents at the two ends take different directions.

The cross-validated q2 of a model of this series depends strongly on which molecule is the reference of the alignment, so a value above a published one is not in itself a finding. Compounds predicted again from unaligned structures through a saved model were returned with their fitted values, which shows that the recorded alignment is replayed exactly when a model is applied to new compounds.

3D structure generation and geometry quality

The 3D structure generator was tested on 417 molecules. All were built with their constitution and all 737 specified stereocentres intact, and no pair of non-bonded heavy atoms came closer than 2.4 Å.

QuantityGeinQSAR StudioReference
Saturated six-membered rings built as chairs (of 107)8682 (lowest-energy conformers of the reference's distance-geometry method with MMFF94)
Median deviation from the nearest reference conformer, up to two rotatable bonds0.2 Å
Median deviation from the nearest reference conformer, three to five rotatable bonds0.5 Å

Contour surfaces are extracted from a spline-refined lattice with gradient normals; for a Gaussian lobe on a 2 Å lattice the enclosed volume is within 0.11 % of the exact value, against 20.7 % for direct contouring of the lattice.

Known limitations