Geinium
Home › Accuracy and validation

Accuracy and validation

How each part of Geinium was validated, against what, and with which result.

Every part of Geinium that produces a number was compared with an independent reference before release: the engine with the double-precision build of an established reference program, the analyses with the same program's analysis tools, MM-GBSA with an established implementation of the method, the ligand chemistry with the reference chain it reproduces, and the interaction analysis with independent tools. Each result below gives the system, the quantity, the tolerance that was set before the comparison, and the largest difference found, not an average. No tolerance was loosened to obtain a pass.

The validations were run between 5 and 9 October 2026 on the development PC, which has one graphics card (see Performance). The GPU engine has been executed on that card only. On another PC, run mdsim gpuinfo -selftest once: it repeats the GPU comparison of this page on a built-in system in about two seconds.

Engine energies and forces

Energies, forces and virials of the CPU engine were compared term by term with the double-precision build of an established reference program on 42 test systems: regression cases, lysozyme (33,752 atoms) in four force-field families, a membrane, virtual sites, and every bonded and restraint type. The topologies were taken from the reference program itself, so the force calculation was tested independently of the rest of Geinium.

ComparisonToleranceLargest difference
Every energy term, 42 systems (2468 comparisons, 2786 test assertions, 0 failed)1e-6 relative5.9e-12 relative
Forces, relative RMS1e-63.4e-13
Virial and pressure1e-61.6e-12
Production force field: ff14SB protein with a GAFF2/AM1-BCC ligand, 33,740 atoms1e-61.0e-12 (potential energy)
Production force field: Lipid21 POPC/cholesterol bilayer, 47,232 atoms1e-64.1e-13
The same calculation repeated 20 times with 8 threadsidentical0 mismatches

Against the mixed-precision build of the reference, 41 of the comparisons differ by more than 1e-5. In each of them the reference's own double-precision build is just as far from its mixed-precision build: the difference is the rounding of single precision in the reference, which is why the double-precision build is the acceptance reference.

Dynamics step by step

The integrator, the constraints and the coupling of the CPU engine were compared with the double-precision reference at every step of 24 short runs: constant energy, V-rescale, Berendsen and Nose-Hoover thermostats, C-rescale and Parrinello-Rahman barostats (isotropic, semi-isotropic, anisotropic), position restraints, centre-of-mass removal, stochastic dynamics, 4 fs with hydrogen-mass repartitioning, and lysozyme with ff14SB. Positions agreed within 1e-6 nm up to 0.2 ps and every energy term within 1e-8 relative; 0 of 91,214 checks failed. The files written by mdsim mdrun differ from the reference files by at most one unit in the last printed digit. Beyond about 0.3 ps two runs separate, as two runs of any MD program do, because molecular dynamics is chaotic (the measured Lyapunov exponent is 11 to 19 per ps); from then on the comparison is statistical, below.

GPU against CPU

The GPU engine computes in mixed precision: single-precision coordinates and pair forces, 64-bit fixed-point force accumulation, and double-precision energies and virial. It was compared with the double-precision CPU engine on the same coordinates.

SystemForces, relative RMSOther quantities
Lysozyme, 33,752 atoms, 2 fs1.295e-6Potential energy 2.1e-9 relative
A 67,156-atom protein-ligand system1.663e-6Potential energy 1.5e-8 relative
Water box, 6,495 atoms8.1e-7
Over a 20-step runLargest deviation of any energy term 2.5e-5 relative (lysozyme) and 1.0e-4 (a membrane protein system); pressure within about 3 bar; temperature within 7e-3 K

Directly against the double-precision reference program, on 13 test systems: potential energy within 2e-9 to 2.1e-7 relative, worst single term 4.8e-6 (tolerance 2e-5), forces 3.1e-6 to 2.4e-5 relative RMS in 11 systems (tolerance 5e-5). For scale, the reference program's own mixed-precision build differs from its double-precision build by 1.6e-5 on lysozyme. Two systems were recorded without a force check because single-precision coordinates decide them: plain cut-off electrostatics (pairs exactly at the cut-off) and water without constraints. The production protocol uses neither.

The test suite of the GPU engine requires forces within 5e-5 relative RMS, every energy term within 5e-6 at the first step and 2e-4 over the run, and a repeated run identical bit for bit. Multiple time stepping (factor 2) stays within these limits: first frame 8e-7 to 4e-6, all frames at most 7.1e-5, pressure within 2.7 bar.

The self test

mdsim gpuinfo -selftest runs a built-in system of 27 ethanol molecules in 1512 water molecules (4,779 atoms, PME, SETTLE and LINCS) for 20 steps, twice on every usable GPU and once on the CPU engine, and applies the tolerances of the test suite. On the development card: forces 8.7e-7 to 9.1e-7, energy terms 3.9e-7 to 7.4e-7 at the first step and 2.5e-5 to 4.4e-5 over the run, the repeated run identical, in 1.5 s. A failure prints "RESULT: FAILED" and returns exit code 1.

Continuation and reproducibility

CheckResult
GPU engine: the same run twiceEnergy, trajectory and final-structure files byte-identical
GPU engine: stopped and continued from the checkpoint; process killed and continuedByte-identical to the uninterrupted run
GPU engine: this release against release 1.0.0Byte-identical output files; the speed-up of this release changed no number
CPU engine, one thread: stopped or killed and continuedByte-identical
CPU engine, eight threads: stopped or killed and continuedTrajectory and final structure identical; the energy file can differ in the last bits
Replicas run side by side on one GPU against one after the other12 of 12 files identical, in 3 of 3 repetitions

Energy conservation

Drift of the total energy in constant-energy runs of lysozyme (33,752 atoms, 50 ps, slope in five blocks of 10 ps, mean and standard error, kJ/mol/ps per atom), against the GPU and CPU builds of the reference program on the same input:

ProtocolGeinium, GPU engineReference, GPU buildReference, CPU build
2 fs (the default)+8.0e-5 ± 0.6e-5+1.18e-4 ± 0.26e-4+6.5e-5 ± 0.7e-5
4 fs, repartitioned hydrogen masses, two LINCS iterations (the pipeline setting)-0.6e-5 ± 5.0e-5+7.8e-5 ± 1.5e-5-3.8e-5 ± 3.7e-5
4 fs, repartitioned hydrogen masses, one LINCS iteration+1.76e-3 ± 0.30e-3+1.43e-3 ± 0.31e-3+8.2e-4 ± 1.2e-4

At 2 fs the GPU engine conserves energy as well as the reference. The last row is the reason the pipeline always uses two LINCS iterations at 4 fs: with one, every program drifts about twenty times more. Multiple time stepping with factor 2 adds no measurable drift (+1.6e-5 ± 1.0e-5 at 2 fs, zero within ±5e-5 at 4 fs).

Ensemble averages

Averages of a thermostatted, barostatted run are the quantity a user actually reports. They were compared with eight independent 1 ns runs of each program (lysozyme, 2 fs, NPT, the runs differing only in the seed of the thermostat), averaged over 20 to 1000 ps; mean and standard error over the eight runs:

QuantityGeiniumReference programDifference in combined standard errors
Temperature (K)299.9973 ± 0.0088299.9994 ± 0.0082-0.18
Pressure (bar)2.54 ± 1.462.80 ± 1.41-0.13
Density (kg/m3)1011.958 ± 0.0961012.035 ± 0.081-0.62
Potential energy (kJ/mol)-469699.1 ± 12.0-469716.6 ± 5.9+1.31 (17.5 kJ/mol, 3.7e-5 relative)

All four agree within statistical error. Replicas were needed for this: a single pair of 1 ns runs had given a potential-energy difference of 2.1 block standard errors, because block errors of one trajectory underestimate the error of a slowly fluctuating protein energy. One observation remains open: the spread of Geinium's eight potential-energy averages is twice that of the reference (33.9 against 16.8 kJ/mol). With eight runs each this is not significant (variance ratio 4.1, below the 5 % limit of 4.99), but it is stated here. Shorter NPT comparisons with multiple time stepping and with 4 fs agreed within 1.5 combined standard errors in all 12 averages.

Ligand parameters

Geinium computes GAFF2 atom types and AM1-BCC charges with its own code, which reproduces an established reference chain stage by stage. Test sets: 44 drug-like molecules (H, C, N, O, F, P, S, Cl, Br, I; charges -1, 0, +1) and 54 small molecules with unusual groups.

ComparisonToleranceResult
AM1 heat of formation, 88 single points0.01 kcal/mol2.8e-8 kcal/mol
AM1 Mulliken charges, the same1e-3 e5.0e-4 e (the rounding of the reference output)
AM1 geometry optimisation, 44 molecules, heat of formation0.01 kcal/mol2.4e-4 kcal/mol
Bond types and GAFF2 atom types, 98 molecules0 mismatches0
AM1-BCC charges from the same AM1 charges, 98 molecules00
Missing-parameter files, 98 moleculesidentical98 of 98
End to end from SDF, 44 molecules: topology lines; chargesidentical; 2e-3 e44 of 44; 1.0e-3 e
The same topologies processed into run inputsaccepted, 0 warnings44 of 44
End to end, 54 unusual moleculesheat of formation 0.01 kcal/mol53 of 54; one oxime 0.052 kcal/mol lower, because the reference ends on a saddle point

The one deliberate difference from the reference chain: a torsion term for which no analogue exists gets an estimated parameter and a warning, where the reference would stop. Parameters taken by analogy are flagged with their penalty (above 50 a note, above 150 a warning); such terms are not validated for that chemistry. See Ligand parameters.

Protonation of ligands

The charge state Geinium assigns at pH 7.4 was compared with curated literature charge states of 362 drug-like molecules: 353 of 362 agree (97.5 %); 315 of 318 clear cases (99.1 %) and 38 of 44 borderline cases (86.4 %, a group within one pKa unit of 7.4 or disputed literature values). Two established protonation tools scored 77.1 % and 83.7 % on the same set. The result does not depend on how the input is protonated: the same charge came out of four input forms for 361 of 362 molecules.

This is a development set, not a blind test: the reference charges were curated during the work, and the rules were refined after a first comparison (which gave 350 of 362). Expect lower agreement on chemistry unlike the set. Groups within about one pKa unit of the pH can go either way; check the protonation message of every ligand.

Analyses

AnalysisCompared withResult
RMSD, RMSF, radius of gyration, energy termsThe analysis tools of the reference program, mixed and double precisionData lines identical (largest difference 0)
Trajectory conversion (whole molecules, centring, fitting)The same158 of 159 files byte-identical in mixed precision, 159 of 159 in double precision
Secondary structure (DSSP method), hydrogen bonds, SASAThe same, on lysozyme and trypsin with benzamidine581 cases, 1,905 files, 921,192 lines: all identical
Protein-ligand hydrogen bonds per frame; donor-acceptor distances; occupanciesIndependent analyses with the reference tools61 of 61 frames identical; distances within 0.0005 nm; fractions within 4.6e-5
C-alpha RMSF of the results exportAn independent recomputationWithin 6e-5 nm (100 ns job) and 5e-5 nm (three 1 ns replicas)
PCA eigenvaluesAn independent numerical library on the same coordinatesWithin 3.8e-7 relative
MM-GBSA statistics, quartiles, correlation matrixAn independent statistics libraryWithin 5.0e-5 kcal/mol
Every number of the report tablesThe raw-data files of the package210 of 210, 220 of 220 and 882 of 882 in the release exports; checked again at every export

Open point: the signs of eigenvectors of the covariance analysis can differ from the reference. The sign of an eigenvector is arbitrary; it flips the axis of a projection, not its content.

Interactions

The interaction detection was compared with independent tools on two 100 ns complexes (1,001 frames each), first with the same definition on both sides, which tests the detection itself:

InteractionGeinium / independent tool / both, complex 1Complex 2
Hydrogen-bond triplets93 triplets, identical frame counts for 93 of 937 of 7
Hydrophobic contacts (residue-frames)799 / 799 / 7991879 / 1879 / 1879
Ionic, ligand anion390 / 390 / 390none on either side
Pi-cation12 / 12 / 12none on either side
Pi stacking, parallel20 / 20 / 180 / 1 / 0
Pi stacking, T-shaped26 / 29 / 2665 / 61 / 56

Every stacking difference is a threshold case: the deciding angle or offset lies within the range spanned by the choice of ring atoms that define the ring plane. On constructed geometries, 23 of 24 decisions agree; the one difference is exactly parallel rings (angle 0), which Geinium accepts and the other tool rejects. With each tool's own definitions the numbers differ, and every difference traced back to a difference of definition. The perception of the ligand chemistry was compared with a chemistry toolkit on 375 molecules: ring sets 375 of 375; charges, donors, acceptors and the hydrophobic, halogen and metal flags of 8,917 of 8,917 atoms; R/S labels 305 of 305.

The real trajectories compared were of anionic ligands. Cationic ligands, ligand hydrogen-bond donors, halogen bonds, metal sites and several ligand copies are covered by constructed test geometries only.

2D structures

The two-dimensional structure drawn for the ligand was checked on a set of 2,634 molecules against reference perceptions: the drawn graph is identical for 2632 of 2633 drawn molecules; 27,018 atom labels were checked with none wrong or missing; all 1447 reference E/Z double bonds are drawn correctly; of 15,335 stereocentres, 133 are not drawn (protonated amines, allenes and other documented classes) and 1 is read inverted. 2405 drawings are free of crossings and overlaps, against 2134 for a widely used chemistry toolkit on the same set. From MOL2 input without bond orders, 2525 were identical, 16 differed (10 of them sulfoxides drawn as S=O) and 20 were refused with a message. CIP rules 3 to 5 are not implemented; an SD file avoids the bond-order problems.

MM-GBSA

MM-GBSA was compared frame by frame with an established implementation of the method, given the same topology and the same trajectory: trypsin with benzamidine (3,238 solute atoms, 101 frames) for all five generalised Born models (HCT, OBC1, OBC2, GBn, GBn2).

QuantityTargetLargest difference per frame
van der Waals, electrostatic and polar solvation terms of the binding energy0.01 kcal/mol1.7e-4 kcal/mol
Non-polar solvation term0.05 kcal/molbelow 1e-5 kcal/mol
Binding energy (total)0.05 kcal/mol5.3e-4 kcal/mol, that is better than 0.001 kcal/mol
Mean, standard deviation and standard error of the total0.005 kcal/molbelow 0.005
Lysozyme with a ligand, production-style topology (1,986 solute atoms, 21 frames), total0.01 kcal/mol2.5e-4 kcal/mol
Per-residue decomposition, 224 residues, mean per residue0.05 kcal/mol4.9e-3 kcal/mol
Bonded terms against the reference MD program1e-6 relativewithin 1e-6

When the reference implementation is run with its default rounded intermediate files, its own per-frame values change by up to 0.06 kcal/mol; the means still agree within 0.0005 kcal/mol. This is a property of the reference. What the agreement means: Geinium computes the method exactly. What it does not mean: the method has no entropy term and no implicit membrane, and its values rank ligands bound to the same receptor; they are not absolute binding free energies (see Binding free energy).

Missing residues and side chains

Gap modelling was tested by removing 2, 4 and 8 residues at three sites in each of five protein structures (45 gaps) and modelling them back. All 45 were closed and accepted by topology building, run-input processing and 500 steps of minimisation; the closing peptide bond was within 0.0043 nm of its ideal length.

Gap lengthBackbone RMSD to the crystal loop, mean (median)Ramachandran favoured / allowed / outlierTime per gap
2 residues0.59 Å (0.33)23 / 6 / 11.5 s
4 residues0.66 Å (0.52)54 / 6 / 03.0 s
8 residues3.11 Å (2.82; largest 6.39)93 / 23 / 420.5 s

Short gaps come back close to the crystal structure. An 8-residue loop is a plausible, clash-free closure, not a prediction of the native loop; treat its conformation, and contacts it makes, with care. Rebuilt side chains reproduce the crystal chi1 angle within 40 degrees in 79 % of cases, chi1 and chi2 together in 68 %.

Membrane orientation

With an oriented reference structure, the protein is superposed on it. Without one, a hydrophobic-slab fit places it; this was within 3.5 degrees and 0.12 nm for a 544-residue transporter but 20.6 degrees off for a small helix dimer. A reference structure is therefore strongly recommended, and the program warns when it is missing.

What was not validated

The detailed records of these validations, with the scripts and reference data, are kept with the program's source; they are not part of the installation.