Accuracy and validation
How each part of Geinium was validated, against what, and with which result.
Every part of Geinium that produces a number was compared with an independent reference before release: the engine with the double-precision build of an established reference program, the analyses with the same program's analysis tools, MM-GBSA with an established implementation of the method, the ligand chemistry with the reference chain it reproduces, and the interaction analysis with independent tools. Each result below gives the system, the quantity, the tolerance that was set before the comparison, and the largest difference found, not an average. No tolerance was loosened to obtain a pass.
mdsim gpuinfo -selftest once: it repeats the GPU
comparison of this page on a built-in system in about two seconds.Engine energies and forces
Energies, forces and virials of the CPU engine were compared term by term with the double-precision build of an established reference program on 42 test systems: regression cases, lysozyme (33,752 atoms) in four force-field families, a membrane, virtual sites, and every bonded and restraint type. The topologies were taken from the reference program itself, so the force calculation was tested independently of the rest of Geinium.
| Comparison | Tolerance | Largest difference |
|---|---|---|
| Every energy term, 42 systems (2468 comparisons, 2786 test assertions, 0 failed) | 1e-6 relative | 5.9e-12 relative |
| Forces, relative RMS | 1e-6 | 3.4e-13 |
| Virial and pressure | 1e-6 | 1.6e-12 |
| Production force field: ff14SB protein with a GAFF2/AM1-BCC ligand, 33,740 atoms | 1e-6 | 1.0e-12 (potential energy) |
| Production force field: Lipid21 POPC/cholesterol bilayer, 47,232 atoms | 1e-6 | 4.1e-13 |
| The same calculation repeated 20 times with 8 threads | identical | 0 mismatches |
Against the mixed-precision build of the reference, 41 of the comparisons differ by more than 1e-5. In each of them the reference's own double-precision build is just as far from its mixed-precision build: the difference is the rounding of single precision in the reference, which is why the double-precision build is the acceptance reference.
Dynamics step by step
The integrator, the constraints and the coupling of the CPU engine were compared with the
double-precision reference at every step of 24 short runs: constant energy, V-rescale, Berendsen
and Nose-Hoover thermostats, C-rescale and Parrinello-Rahman barostats (isotropic, semi-isotropic,
anisotropic), position restraints, centre-of-mass removal, stochastic dynamics, 4 fs with
hydrogen-mass repartitioning, and lysozyme with ff14SB. Positions agreed within 1e-6 nm up to
0.2 ps and every energy term within 1e-8 relative; 0 of 91,214 checks failed. The files written by
mdsim mdrun differ from the reference files by at most one unit in the last printed
digit. Beyond about 0.3 ps two runs separate, as two runs of any MD program do, because molecular
dynamics is chaotic (the measured Lyapunov exponent is 11 to 19 per ps); from then on the
comparison is statistical, below.
GPU against CPU
The GPU engine computes in mixed precision: single-precision coordinates and pair forces, 64-bit fixed-point force accumulation, and double-precision energies and virial. It was compared with the double-precision CPU engine on the same coordinates.
| System | Forces, relative RMS | Other quantities |
|---|---|---|
| Lysozyme, 33,752 atoms, 2 fs | 1.295e-6 | Potential energy 2.1e-9 relative |
| A 67,156-atom protein-ligand system | 1.663e-6 | Potential energy 1.5e-8 relative |
| Water box, 6,495 atoms | 8.1e-7 | |
| Over a 20-step run | Largest deviation of any energy term 2.5e-5 relative (lysozyme) and 1.0e-4 (a membrane protein system); pressure within about 3 bar; temperature within 7e-3 K |
Directly against the double-precision reference program, on 13 test systems: potential energy within 2e-9 to 2.1e-7 relative, worst single term 4.8e-6 (tolerance 2e-5), forces 3.1e-6 to 2.4e-5 relative RMS in 11 systems (tolerance 5e-5). For scale, the reference program's own mixed-precision build differs from its double-precision build by 1.6e-5 on lysozyme. Two systems were recorded without a force check because single-precision coordinates decide them: plain cut-off electrostatics (pairs exactly at the cut-off) and water without constraints. The production protocol uses neither.
The test suite of the GPU engine requires forces within 5e-5 relative RMS, every energy term within 5e-6 at the first step and 2e-4 over the run, and a repeated run identical bit for bit. Multiple time stepping (factor 2) stays within these limits: first frame 8e-7 to 4e-6, all frames at most 7.1e-5, pressure within 2.7 bar.
The self test
mdsim gpuinfo -selftest runs a built-in system of 27 ethanol molecules in 1512 water
molecules (4,779 atoms, PME, SETTLE and LINCS) for 20 steps, twice on every usable GPU and once on the
CPU engine, and applies the tolerances of the test suite. On the development card: forces 8.7e-7 to
9.1e-7, energy terms 3.9e-7 to 7.4e-7 at the first step and 2.5e-5 to 4.4e-5 over the run, the
repeated run identical, in 1.5 s. A failure prints "RESULT: FAILED" and returns exit code 1.
Continuation and reproducibility
| Check | Result |
|---|---|
| GPU engine: the same run twice | Energy, trajectory and final-structure files byte-identical |
| GPU engine: stopped and continued from the checkpoint; process killed and continued | Byte-identical to the uninterrupted run |
| GPU engine: this release against release 1.0.0 | Byte-identical output files; the speed-up of this release changed no number |
| CPU engine, one thread: stopped or killed and continued | Byte-identical |
| CPU engine, eight threads: stopped or killed and continued | Trajectory and final structure identical; the energy file can differ in the last bits |
| Replicas run side by side on one GPU against one after the other | 12 of 12 files identical, in 3 of 3 repetitions |
Energy conservation
Drift of the total energy in constant-energy runs of lysozyme (33,752 atoms, 50 ps, slope in five blocks of 10 ps, mean and standard error, kJ/mol/ps per atom), against the GPU and CPU builds of the reference program on the same input:
| Protocol | Geinium, GPU engine | Reference, GPU build | Reference, CPU build |
|---|---|---|---|
| 2 fs (the default) | +8.0e-5 ± 0.6e-5 | +1.18e-4 ± 0.26e-4 | +6.5e-5 ± 0.7e-5 |
| 4 fs, repartitioned hydrogen masses, two LINCS iterations (the pipeline setting) | -0.6e-5 ± 5.0e-5 | +7.8e-5 ± 1.5e-5 | -3.8e-5 ± 3.7e-5 |
| 4 fs, repartitioned hydrogen masses, one LINCS iteration | +1.76e-3 ± 0.30e-3 | +1.43e-3 ± 0.31e-3 | +8.2e-4 ± 1.2e-4 |
At 2 fs the GPU engine conserves energy as well as the reference. The last row is the reason the pipeline always uses two LINCS iterations at 4 fs: with one, every program drifts about twenty times more. Multiple time stepping with factor 2 adds no measurable drift (+1.6e-5 ± 1.0e-5 at 2 fs, zero within ±5e-5 at 4 fs).
Ensemble averages
Averages of a thermostatted, barostatted run are the quantity a user actually reports. They were compared with eight independent 1 ns runs of each program (lysozyme, 2 fs, NPT, the runs differing only in the seed of the thermostat), averaged over 20 to 1000 ps; mean and standard error over the eight runs:
| Quantity | Geinium | Reference program | Difference in combined standard errors |
|---|---|---|---|
| Temperature (K) | 299.9973 ± 0.0088 | 299.9994 ± 0.0082 | -0.18 |
| Pressure (bar) | 2.54 ± 1.46 | 2.80 ± 1.41 | -0.13 |
| Density (kg/m3) | 1011.958 ± 0.096 | 1012.035 ± 0.081 | -0.62 |
| Potential energy (kJ/mol) | -469699.1 ± 12.0 | -469716.6 ± 5.9 | +1.31 (17.5 kJ/mol, 3.7e-5 relative) |
All four agree within statistical error. Replicas were needed for this: a single pair of 1 ns runs had given a potential-energy difference of 2.1 block standard errors, because block errors of one trajectory underestimate the error of a slowly fluctuating protein energy. One observation remains open: the spread of Geinium's eight potential-energy averages is twice that of the reference (33.9 against 16.8 kJ/mol). With eight runs each this is not significant (variance ratio 4.1, below the 5 % limit of 4.99), but it is stated here. Shorter NPT comparisons with multiple time stepping and with 4 fs agreed within 1.5 combined standard errors in all 12 averages.
Ligand parameters
Geinium computes GAFF2 atom types and AM1-BCC charges with its own code, which reproduces an established reference chain stage by stage. Test sets: 44 drug-like molecules (H, C, N, O, F, P, S, Cl, Br, I; charges -1, 0, +1) and 54 small molecules with unusual groups.
| Comparison | Tolerance | Result |
|---|---|---|
| AM1 heat of formation, 88 single points | 0.01 kcal/mol | 2.8e-8 kcal/mol |
| AM1 Mulliken charges, the same | 1e-3 e | 5.0e-4 e (the rounding of the reference output) |
| AM1 geometry optimisation, 44 molecules, heat of formation | 0.01 kcal/mol | 2.4e-4 kcal/mol |
| Bond types and GAFF2 atom types, 98 molecules | 0 mismatches | 0 |
| AM1-BCC charges from the same AM1 charges, 98 molecules | 0 | 0 |
| Missing-parameter files, 98 molecules | identical | 98 of 98 |
| End to end from SDF, 44 molecules: topology lines; charges | identical; 2e-3 e | 44 of 44; 1.0e-3 e |
| The same topologies processed into run inputs | accepted, 0 warnings | 44 of 44 |
| End to end, 54 unusual molecules | heat of formation 0.01 kcal/mol | 53 of 54; one oxime 0.052 kcal/mol lower, because the reference ends on a saddle point |
The one deliberate difference from the reference chain: a torsion term for which no analogue exists gets an estimated parameter and a warning, where the reference would stop. Parameters taken by analogy are flagged with their penalty (above 50 a note, above 150 a warning); such terms are not validated for that chemistry. See Ligand parameters.
Protonation of ligands
The charge state Geinium assigns at pH 7.4 was compared with curated literature charge states of 362 drug-like molecules: 353 of 362 agree (97.5 %); 315 of 318 clear cases (99.1 %) and 38 of 44 borderline cases (86.4 %, a group within one pKa unit of 7.4 or disputed literature values). Two established protonation tools scored 77.1 % and 83.7 % on the same set. The result does not depend on how the input is protonated: the same charge came out of four input forms for 361 of 362 molecules.
Analyses
| Analysis | Compared with | Result |
|---|---|---|
| RMSD, RMSF, radius of gyration, energy terms | The analysis tools of the reference program, mixed and double precision | Data lines identical (largest difference 0) |
| Trajectory conversion (whole molecules, centring, fitting) | The same | 158 of 159 files byte-identical in mixed precision, 159 of 159 in double precision |
| Secondary structure (DSSP method), hydrogen bonds, SASA | The same, on lysozyme and trypsin with benzamidine | 581 cases, 1,905 files, 921,192 lines: all identical |
| Protein-ligand hydrogen bonds per frame; donor-acceptor distances; occupancies | Independent analyses with the reference tools | 61 of 61 frames identical; distances within 0.0005 nm; fractions within 4.6e-5 |
| C-alpha RMSF of the results export | An independent recomputation | Within 6e-5 nm (100 ns job) and 5e-5 nm (three 1 ns replicas) |
| PCA eigenvalues | An independent numerical library on the same coordinates | Within 3.8e-7 relative |
| MM-GBSA statistics, quartiles, correlation matrix | An independent statistics library | Within 5.0e-5 kcal/mol |
| Every number of the report tables | The raw-data files of the package | 210 of 210, 220 of 220 and 882 of 882 in the release exports; checked again at every export |
Open point: the signs of eigenvectors of the covariance analysis can differ from the reference. The sign of an eigenvector is arbitrary; it flips the axis of a projection, not its content.
Interactions
The interaction detection was compared with independent tools on two 100 ns complexes (1,001 frames each), first with the same definition on both sides, which tests the detection itself:
| Interaction | Geinium / independent tool / both, complex 1 | Complex 2 |
|---|---|---|
| Hydrogen-bond triplets | 93 triplets, identical frame counts for 93 of 93 | 7 of 7 |
| Hydrophobic contacts (residue-frames) | 799 / 799 / 799 | 1879 / 1879 / 1879 |
| Ionic, ligand anion | 390 / 390 / 390 | none on either side |
| Pi-cation | 12 / 12 / 12 | none on either side |
| Pi stacking, parallel | 20 / 20 / 18 | 0 / 1 / 0 |
| Pi stacking, T-shaped | 26 / 29 / 26 | 65 / 61 / 56 |
Every stacking difference is a threshold case: the deciding angle or offset lies within the range spanned by the choice of ring atoms that define the ring plane. On constructed geometries, 23 of 24 decisions agree; the one difference is exactly parallel rings (angle 0), which Geinium accepts and the other tool rejects. With each tool's own definitions the numbers differ, and every difference traced back to a difference of definition. The perception of the ligand chemistry was compared with a chemistry toolkit on 375 molecules: ring sets 375 of 375; charges, donors, acceptors and the hydrophobic, halogen and metal flags of 8,917 of 8,917 atoms; R/S labels 305 of 305.
2D structures
The two-dimensional structure drawn for the ligand was checked on a set of 2,634 molecules against reference perceptions: the drawn graph is identical for 2632 of 2633 drawn molecules; 27,018 atom labels were checked with none wrong or missing; all 1447 reference E/Z double bonds are drawn correctly; of 15,335 stereocentres, 133 are not drawn (protonated amines, allenes and other documented classes) and 1 is read inverted. 2405 drawings are free of crossings and overlaps, against 2134 for a widely used chemistry toolkit on the same set. From MOL2 input without bond orders, 2525 were identical, 16 differed (10 of them sulfoxides drawn as S=O) and 20 were refused with a message. CIP rules 3 to 5 are not implemented; an SD file avoids the bond-order problems.
MM-GBSA
MM-GBSA was compared frame by frame with an established implementation of the method, given the same topology and the same trajectory: trypsin with benzamidine (3,238 solute atoms, 101 frames) for all five generalised Born models (HCT, OBC1, OBC2, GBn, GBn2).
| Quantity | Target | Largest difference per frame |
|---|---|---|
| van der Waals, electrostatic and polar solvation terms of the binding energy | 0.01 kcal/mol | 1.7e-4 kcal/mol |
| Non-polar solvation term | 0.05 kcal/mol | below 1e-5 kcal/mol |
| Binding energy (total) | 0.05 kcal/mol | 5.3e-4 kcal/mol, that is better than 0.001 kcal/mol |
| Mean, standard deviation and standard error of the total | 0.005 kcal/mol | below 0.005 |
| Lysozyme with a ligand, production-style topology (1,986 solute atoms, 21 frames), total | 0.01 kcal/mol | 2.5e-4 kcal/mol |
| Per-residue decomposition, 224 residues, mean per residue | 0.05 kcal/mol | 4.9e-3 kcal/mol |
| Bonded terms against the reference MD program | 1e-6 relative | within 1e-6 |
When the reference implementation is run with its default rounded intermediate files, its own per-frame values change by up to 0.06 kcal/mol; the means still agree within 0.0005 kcal/mol. This is a property of the reference. What the agreement means: Geinium computes the method exactly. What it does not mean: the method has no entropy term and no implicit membrane, and its values rank ligands bound to the same receptor; they are not absolute binding free energies (see Binding free energy).
Missing residues and side chains
Gap modelling was tested by removing 2, 4 and 8 residues at three sites in each of five protein structures (45 gaps) and modelling them back. All 45 were closed and accepted by topology building, run-input processing and 500 steps of minimisation; the closing peptide bond was within 0.0043 nm of its ideal length.
| Gap length | Backbone RMSD to the crystal loop, mean (median) | Ramachandran favoured / allowed / outlier | Time per gap |
|---|---|---|---|
| 2 residues | 0.59 Å (0.33) | 23 / 6 / 1 | 1.5 s |
| 4 residues | 0.66 Å (0.52) | 54 / 6 / 0 | 3.0 s |
| 8 residues | 3.11 Å (2.82; largest 6.39) | 93 / 23 / 4 | 20.5 s |
Short gaps come back close to the crystal structure. An 8-residue loop is a plausible, clash-free closure, not a prediction of the native loop; treat its conformation, and contacts it makes, with care. Rebuilt side chains reproduce the crystal chi1 angle within 40 degrees in 79 % of cases, chi1 and chi2 together in 68 %.
Membrane orientation
With an oriented reference structure, the protein is superposed on it. Without one, a hydrophobic-slab fit places it; this was within 3.5 degrees and 0.12 nm for a 544-residue transporter but 20.6 degrees off for a small helix dimer. A reference structure is therefore strongly recommended, and the program warns when it is missing.
What was not validated
- The GPU engine on any card other than the development card, and builds for other toolkit generations.
- Long-time behaviour beyond the 1 ns ensemble comparisons above; for example the equilibration of a POPC:cholesterol bilayer, which needs 200 ns or more of unrestrained simulation (the built-in membrane protocol equilibrates for 8.5 ns).
- MM-GBSA for membrane systems, the interaction-entropy option (formula test only), and force fields other than AMBER forms.
- Interaction types that occurred only in constructed geometries (see above).
- Ligand torsion parameters taken by analogy with a high penalty.
- Protonation rules on a blind test set.
- Free-energy methods: Geinium has no free-energy perturbation, and the GPU engine refuses such inputs.
The detailed records of these validations, with the scripts and reference data, are kept with the program's source; they are not part of the installation.