GeinQSAR Studio
Home › The model dashboard

The model dashboard

The report of a regression model: the acceptance criteria, the validation figures, the descriptors, the applicability domain and every prediction, read from the run that built it.

The model dashboard opens when a regression workflow finishes, whether the workflow came from the 2D-QSAR wizard, a template or the canvas. It does not compute anything of its own. Every number and every figure is read from the output of the validation node of the workflow, so the dashboard, the exported report and the files the workflow wrote always agree. A workflow that validates several models side by side opens one dashboard per model, with the primary model in front.

The dashboard has five tabs: Overview, Validation, Descriptors, Domain and Compounds. Export report at the right of the tab bar writes the report of the run as PDF, Word or Excel (see Reports and export). The inspector on the right summarises the model: its type, the number of training, test and external compounds, Q2, the Y-randomisation statistic with its verdict, the compounds you have selected with their observed and predicted values, and the descriptors the model uses.

ControlWhat it does
CardOne statistic. A card with an acceptance threshold shows the threshold and a PASS or FAIL badge, and a coloured bar on its left edge: green for a pass, red for a fail.
Help markThe small circled i next to a heading. Its tooltip explains the section and, for a figure, what the run says about it (the space the leverage was computed in, the value of h*).
Camera buttonAt the right of every chart. Opens the figure dialog for that chart (see Publication figures).
Save as CSVAt the right of a table heading. Writes the table as the run produced it, at full precision.
Tooltip of a cardThe definition of the statistic and the reference its threshold comes from.

Overview: the acceptance criteria

The first section, Acceptance criteria, is one card per statistic of the published validation criteria for regression QSAR models. A criterion that fails is reported as measured; the dashboard does not hide or soften it. Read the cards in three groups: goodness of fit (R2 of the training set), robustness (the cross-validated Q2 and the Y-randomisation statistic) and predictivity (everything computed on the test set). A model is judged on the last two groups. The training R2 has no threshold because a flexible model can always raise it.

The Overview tab of a PLS model on 45 training and 15 test compounds. The cards that measure prediction come first and the training figures follow under a heading of their own, Fit (not a measure of prediction). The cross-validated Q2 is 0.389 and every external criterion fails; only cR2p (from R2) passes. Golbraikh-Tropsha reads 1 of 4 met.
The Overview tab of a PLS model on 45 training and 15 test compounds. The cards that measure prediction come first and the training figures follow under a heading of their own, Fit (not a measure of prediction). The cross-validated Q2 is 0.389 and every external criterion fails; only cR2p (from R2) passes. Golbraikh-Tropsha reads 1 of 4 met.
A random forest on the same compounds, arranged the same way. Its Y-randomisation card reads cR2p (from Q2), the cross-validated form used for a flexible learner, and it fails at 0.266. The test statistics are the better of the two models here, which on 15 test compounds is not a result: see Accuracy and validation.
A random forest on the same compounds, arranged the same way. Its Y-randomisation card reads cR2p (from Q2), the cross-validated form used for a flexible learner, and it fails at 0.266. The test statistics are the better of the two models here, which on 15 test compounds is not a result: see Accuracy and validation.
CardWhat it measuresAccepted whenReference
R2 (training)1 - SS(residual) / SS(total) on the training compounds. Goodness of fit only.no thresholdDraper and Smith, Applied Regression Analysis, 3rd ed., Wiley 1998
Q2 with the scheme as subscript (for example 5-FOLD X 3)1 - PRESS / SS(total), each training compound predicted by a model fitted without it. The subscript names the cross-validation scheme and the repeats.> 0.5Golbraikh and Tropsha, J. Mol. Graph. Model. 20:269 (2002)
R2 (test)R2 of the test compounds, which took no part in the model.> 0.6Golbraikh and Tropsha (2002)
Q2F11 - PRESS(test) / SS of the test values about the training mean. Punishes a test set whose mean differs from the training mean.>= 0.70Shi et al., J. Chem. Inf. Comput. Sci. 41:186 (2001); threshold: Chirico and Gramatica, J. Chem. Inf. Model. 52:2044 (2012)
Q2F21 - PRESS(test) / SS of the test values about their own mean.>= 0.70Schüürmann et al., J. Chem. Inf. Model. 48:2140 (2008)
Q2F31 - [PRESS(test) / n(test)] / [SS(training) / n(training)]: the test error scaled by the spread of the training set, so it does not depend on how the test values happen to be distributed.>= 0.70Consonni, Ballabio and Todeschini, J. Chem. Inf. Model. 49:1669 (2009)
CCC (test)Concordance correlation coefficient: how closely the observed and predicted test values lie on the identity line, not merely on some line.>= 0.85Lin, Biometrics 45:255 (1989); Chirico and Gramatica (2011, 2012)
r2m (average)Mean of r2m and r'2m, where r2m = r2 (1 - sqrt(r2 - r02)) with r02 from the regression through the origin.> 0.5Roy et al., J. Chem. Inf. Model. 52:396 (2012)
Δr2mAbsolute difference of r2m and r'2m for the two axis assignments. A large value says the predictions are biased in scale.< 0.2Roy et al. (2012)
Golbraikh-TropshaCounts how many of the individual test-set conditions are met: R2(test) > 0.6, a slope k or k' of the regression through the origin between 0.85 and 1.15, and (R2 - R02) / R2 < 0.1.all satisfiedGolbraikh and Tropsha (2002)
cR2p (from R2) or (from Q2)R sqrt(R2 - mean R2 of the Y-randomised models), or the same with the cross-validated Q2. A chance correlation gives a value near 0.> 0.5Mitra, Saha and Roy, Mol. Simul. 36:1067 (2010)
RMSEPRoot mean squared error of the test predictions, in the units of the response. The number to quote as the expected error of a prediction.no thresholdDraper and Smith (1998)
A test statistic is no more precise than the number of test compounds allows. A series of about fifty compounds leaves about a dozen in the test set, so its R2 (test) and RMSEP should be read as the right answer plus or minus roughly 30 %. Quote the cross-validated statistics with the spread the dashboard prints, or repeat the split with other seeds and report that spread, rather than one test figure carried to three decimals (see The 2D-QSAR wizard).

What a passing model looks like: the ESOL aqueous solubility set (1128 compounds, 838 training and 279 test, the default recipe with three repeats of 5-fold cross-validation) gives R2 0.977, Q2 0.889, R2 (test) 0.883 and RMSEP 0.727 log units, and passes every card. The PLS model on 60 acetylcholinesterase inhibitors above, with about a thousand descriptor columns, fails nearly all of them. The difference is the data, not the program: 45 training compounds do not support a model on a thousand descriptor columns.

The Overview of the ESOL model. Every card has a green bar and a PASS badge. Below the cards the Model section names the fitted pipeline (PLS with 8 latent variables), the rows it was fitted on and the 3003 descriptors it uses; the equation shows its 12 largest terms.
The Overview of the ESOL model. Every card has a green bar and a PASS badge. Below the cards the Model section names the fitted pipeline (PLS with 8 latent variables), the rows it was fitted on and the 3003 descriptors it uses; the equation shows its 12 largest terms.

Model, equation and All metrics

Under the cards, Model lists what was fitted: the model type, Fitted as (the pipeline with its preprocessing and, for PLS, the number of latent variables), Fitted on (how many training rows), the response, the compounds per set, the number of descriptors, the effective predictors and the latent variables. A linear model also shows its Equation; when it has more terms than fit on the page, the largest are shown and a line says how many there are. The warnings of the run follow, then All metrics: every statistic the validation produced, including those without a card, with the set it belongs to, its value, the criterion, the verdict and the reference. Counts are printed as integers (n = 45) and the Golbraikh-Tropsha row as yes or no. A statistic that could not be computed is left empty, never shown as zero.

All metrics. Each row names its set (Training, Cross-validation, Test), the metric and its value; rows with an acceptance criterion carry the criterion, the verdict and the reference in the columns to the right. Counts such as n and effective predictors are whole numbers.
All metrics. Each row names its set (Training, Cross-validation, Test), the metric and its value; rows with an acceptance criterion carry the criterion, the verdict and the reference in the columns to the right. Counts such as n and effective predictors are whole numbers.

Validation

The Validation tab shows every figure the validation node delivered, each under the title the run gave it, and nothing the run did not produce. A figure that is absent takes no place and prints no heading.

FigureWhat to look for
Observed vs predictedTraining, cross-validated and test predictions as three series against the identity line, with a statistics box (N, R2, RMSE, Q2CV, Q2F1, Q2F3, R2test). The cross-validated points are the honest ones for the training compounds; a cloud of training points tight on the line with the cross-validated points spread around it is the picture of overfitting.
Residuals vs predictedObserved minus predicted against the prediction, with a band of one log unit. A funnel says the error depends on the activity range. A downward trend is expected for every compound the model did not fit, where the slope of the residual on the prediction is about minus var(residual) / var(predicted); the trend itself is read from the residual slope rows of All metrics, which carry that expected value beside the measured one. On the training compounds the same slope measures only how far the fit was shrunk towards the mean: it is exactly zero for least squares and PLS, and positive for ridge, kernel and forest models. All metrics reports it there without an expected value and without a p-value, because it cannot show a fault.
Williams plot, or its replacementSee Domain below. For a model without a hat matrix the figure is the standardised residual against the distance the domain measures, and it is titled accordingly.
Y-randomisationOne point per permuted model: its R2 (and Q2 when computed) against the correlation of the permuted with the true response, and the true model at correlation 1. The permuted points should sit far below the true one.
Observed against predicted for the PLS model. The blue training points follow the identity line closely while the orange cross-validated predictions of the same compounds scatter widely: R2 = 0.896 against Q2CV = 0.389 in the statistics box.
Observed against predicted for the PLS model. The blue training points follow the identity line closely while the orange cross-validated predictions of the same compounds scatter widely: R2 = 0.896 against Q2CV = 0.389 in the statistics box.
The same figure for the random forest. The forest pulls its predictions towards the mean, so the points form a line flatter than the identity line at both ends of the activity range.
The same figure for the random forest. The forest pulls its predictions towards the mean, so the points form a line flatter than the identity line at both ends of the activity range.
The ESOL model exported as a print figure. The three series lie along the identity line over twelve log units; the statistics box gives Q2CV = 0.889 and R2test = 0.883.
The ESOL model exported as a print figure. The three series lie along the identity line over twelve log units; the statistics box gives Q2CV = 0.889 and R2test = 0.883.
Residuals of the ESOL model. Most points lie inside the grey band of one log unit; the test residuals (SD 0.727) are wider than the training residuals (SD 0.316), as they should be for compounds the model did not see.
Residuals of the ESOL model. Most points lie inside the grey band of one log unit; the test residuals (SD 0.727) are wider than the training residuals (SD 0.316), as they should be for compounds the model did not see.

Y-randomisation: which statistic decided

Below the figures the Y-randomization section says on which statistic the verdict was judged (Judged on), its value and verdict, the p value of the permutation test, the number of Permutations and a table with the true model in its first row and one row per permuted run: the correlation of the permuted with the true response, R2 and, when computed, Q2.

Judged onStatisticUsed for
cR2p from the training R2R sqrt(R2 - mean R2 of the permuted models), on the training rowsMLR, stepwise MLR, PLS, PCR, ridge and elastic net
cR2p from the cross-validated Q2Q sqrt(Q2 - mean Q2 of the permuted models), cross-validated on the same folds as the modelevery other model type

A random forest or a support vector machine reproduces scrambled activities on its own training rows, so its training R2 is no evidence against chance correlation; the cross-validated form is. The 0.5 limit was proposed for the training-R2 form and is carried over to the cross-validated one (Rücker, Rücker and Meringer, J. Chem. Inf. Model. 47:2345, 2007, for the method). Read the p value together with the card: it is the share of permutations that scored at least as well as the true model.

Y-randomisation of the PLS model on 45 compounds, judged on cR2p from the training R2: 0.572, PASS. The table tells the rest: several permuted runs reach R2 of 0.79 to 0.99, and the p value is 0.1373. With a thousand descriptor columns and 45 compounds PLS can fit noise.
Y-randomisation of the PLS model on 45 compounds, judged on cR2p from the training R2: 0.572, PASS. The table tells the rest: several permuted runs reach R2 of 0.79 to 0.99, and the p value is 0.1373. With a thousand descriptor columns and 45 compounds PLS can fit noise.
The random forest is judged on cR2p from the cross-validated Q2: 0.266, FAIL. Its permuted runs keep a training R2 near 0.7 but their Q2 is negative, which is why the training form would mislead.
The random forest is judged on cR2p from the cross-validated Q2: 0.266, FAIL. Its permuted runs keep a training R2 near 0.7 but their Q2 is negative, which is why the training form would mislead.
Y-randomisation of the ESOL model as a print figure. The 50 permuted models cluster at R2 0.1 to 0.35 near zero correlation; the true model (red diamond) stands at R2 0.98. The subtitle gives the intercept of the fitted line at zero correlation.
Y-randomisation of the ESOL model as a print figure. The 50 permuted models cluster at R2 0.1 to 0.35 near zero correlation; the true model (red diamond) stands at R2 0.98. The subtitle gives the intercept of the fitted line at zero correlation.

Descriptors

The Descriptors tab shows the Regression coefficients of a linear model, the VIP scores of a PLS model (variable importance in the projection: descriptors with VIP above 1 carry more than their share of the model), the correlation matrix of the model descriptors and a PCA of the descriptor space. These are the mechanistic-interpretation part of the OECD principles.

Regression coefficients of the PLS model on about a thousand columns. The bars form a dense band and the axis labels overlap: for a model this wide, read the leading terms in the equation on the Overview tab or the VIP scores below, and keep this chart for a model with a few descriptors.
Regression coefficients of the PLS model on about a thousand columns. The bars form a dense band and the axis labels overlap: for a model this wide, read the leading terms in the equation on the Overview tab or the VIP scores below, and keep this chart for a model with a few descriptors.

Domain

The applicability domain is the region of descriptor space the training set covers; a prediction outside it is an extrapolation. The domain is fitted on the training compounds together with the model and stored with it, so the verdicts shown here are the ones a later prediction gives.

The Domain tab of the PLS model: 42 of 45 training and all 15 test compounds inside. The method cards show that the kNN distance flags 6.7 % of the training set and the leverage 2.2 %; neither flags a test compound.
The Domain tab of the PLS model: 42 of 45 training and all 15 test compounds inside. The method cards show that the kNN distance flags 6.7 % of the training set and the leverage 2.2 %; neither flags a test compound.
The random forest has no leverage, so only the kNN distance card appears. The distance distribution below is titled with the space it was measured in, the Tanimoto distance to the training set.
The random forest has no leverage, so only the kNN distance card appears. The distance distribution below is titled with the space it was measured in, the Tanimoto distance to the training set.
MethodDefinitionReference
Leverageh = x'(X'X)-1x in the regression space of the model, with the warning leverage h* = 3 (A + 1) / n: A latent components of a PLS or PCR model (or the descriptors of an MLR model) and n training compounds.Gramatica, QSAR Comb. Sci. 26:694 (2007)
Williams plotStandardised residual against leverage. A compound with a residual beyond ±3 is a response outlier; one beyond h* is a structural outlier, predicted by extrapolation.Eriksson et al., Environ. Health Perspect. 111:1361 (2003)
kNN distanceMean distance to the 5 nearest training compounds. The threshold is set from the same distances inside the training set; the default is their 0.95 quantile, and the statistics box of the distance distribution names the rule that was used.Tropsha, Gramatica and Gombar, QSAR Comb. Sci. 22:69 (2003); Sahigara et al., Molecules 17:4791 (2012)
Descriptor range, PCA bounding boxInside the range the training set spans in every descriptor, or in every principal component. Off by default.Sahigara et al. (2012)
DModX, Hotelling T2Distance to the model and distance inside the score space of a latent-variable model, each with its F-based critical value. Off by default.Eriksson et al., Multi- and Megavariate Data Analysis, Umetrics 2006

By default a compound is outside when the leverage or the kNN distance flags it. The rule is set in the wizard (Outside the domain when, see The 2D-QSAR wizard).

The Williams plot of the PLS model, drawn in the space of its 3 components with h* = 0.2667 for n = 45. One training compound lies beyond h*; one test compound has a standardised residual near -6, a response outlier well inside the structural domain.
The Williams plot of the PLS model, drawn in the space of its 3 components with h* = 0.2667 for n = 45. One training compound lies beyond h*; one test compound has a standardised residual near -6, a response outlier well inside the structural domain.
The Williams plot of the ESOL model in the space of 8 PLS components, h* = 0.03222. Two training compounds sit at a leverage near 0.4 with small residuals: they are structurally unusual but well fitted. The test compounds with residuals beyond ±3 lie at low leverage.
The Williams plot of the ESOL model in the space of 8 PLS components, h* = 0.03222. Two training compounds sit at a leverage near 0.4 with small residuals: they are structurally unusual but well fitted. The test compounds with residuals beyond ±3 lie at low leverage.
The distance distribution of the ESOL model as a print figure. The threshold of 0.805 is the 0.95 quantile of the training distances; 11 of 279 test compounds (3.9 %) lie beyond it, in the shaded region outside the domain.
The distance distribution of the ESOL model as a print figure. The threshold of 0.805 is the 0.95 quantile of the training distances; 11 of 279 test compounds (3.9 %) lie beyond it, in the shaded region outside the domain.

Compounds and the linked spreadsheet

The Compounds tab lists every prediction: compound, set, observed, predicted and residual, the cross-validated prediction with its residual and how often the compound was held out, and the domain flag. Save CV predictions writes the cross-validated predictions, the numbers Q2 is computed from.

The compound table and the scatter plots of the dashboard share one selection: select points in a plot and the rows are selected, and the other way round. A permuted model in the Y-randomisation figure selects nothing, because it is not a compound. Show in a spreadsheet opens the predictions as a dataset with structures that shares the same selection; Clear selection empties it.

The Compounds tab. Each row gives the observed value, the fitted prediction and its residual, and the cross-validated prediction: compare Predicted with CV predicted for compound A005 (4.699 against 6.349) to see what the model cannot reproduce once the compound is left out.
The Compounds tab. Each row gives the observed value, the fitted prediction and its residual, and the cross-validated prediction: compare Predicted with CV predicted for compound A005 (4.699 against 6.349) to see what the model cannot reproduce once the compound is left out.
Two compounds selected in the dashboard. The inspector lists them under Selected compounds with their observed and predicted values (A001 6.11 to 6.03).
Two compounds selected in the dashboard. The inspector lists them under Selected compounds with their observed and predicted values (A001 6.11 to 6.03).
The same selection after Show in a spreadsheet: the predictions open as a dataset with structures, and rows 1 and 3 are selected there too.
The same selection after Show in a spreadsheet: the predictions open as a dataset with structures, and rows 1 and 3 are selected there too.

Where the numbers come from

A dashboard belongs to the workflow that produced it. The model type, its hyper-parameters, the preprocessing inside the model and the domain settings are parameters of the Train model node; select the node on the canvas and the inspector shows them. Change one and run the workflow again to get a new dashboard; Workflow > Open results dashboards reopens the dashboards of the last run.

The Train model node selected on the canvas. The inspector shows its parameters: the model (Partial least squares), the response, the imputation, scaling and outlier limits inside the model. The status bar reports that its dataset input is not connected yet, which is why this node would not run.
The Train model node selected on the canvas. The inspector shows its parameters: the model (Partial least squares), the response, the imputation, scaling and outlier limits inside the model. The status bar reports that its dataset input is not connected yet, which is why this node would not run.

Figures for a paper

The camera button of a chart opens the figure dialog: SVG or PNG, the size in millimetres or pixels, the resolution, and Publication figure (170 mm, 600 dpi, the publication theme) in one step. Below 120 mm, a single-column figure, the chart is adapted to the width: smaller text, closer ticks, no statistics box (its numbers belong in the caption) and the legend beside the frame. The figures of this page marked as print figures were written this way at 174 mm and 300 dpi. See Publication figures for every option.

The OECD principles behind this report (a defined endpoint, an unambiguous algorithm, a defined applicability domain, measures of fit, robustness and predictivity, and a mechanistic interpretation where possible) are set out in The 2D-QSAR wizard, where each is tied to the page that settles it.