Geinium
Home › The simulation engine and the GPU

The simulation engine and the GPU

The GPU engine and the CPU reference engine, how one is chosen, how a device is picked, memory, checkpoints, what the GPU engine does not run, and how to test a PC.

Two engines

EnginePrecisionWhere it runsUse
GPU engineMixed: single-precision coordinates and pair forces, 64-bit fixed-point accumulation of forces, double-precision energies and virial.Everything on the device: pair search, non-bonded forces, PME, bonded forces, update, LINCS and SETTLE, temperature and pressure coupling, centre-of-mass removal. Each step is one replayed sequence of GPU work.Every simulation on a PC with a usable NVIDIA GPU.
CPU engineDouble precision throughout.The processor, on several threads (at most 8 for jobs from the window).The reference implementation: the fall-back without a GPU, and always for energy minimisation.

Both engines implement the same algorithms and give the same results within the precision of each. On lysozyme (33,752 atoms) the forces of the GPU engine differed from those of the CPU engine by 1.3e-6 relative RMS and the potential energy by 2.1e-9 relative. Over a 20-step run the largest deviation of any energy term was 2.5e-5 for lysozyme and 1.0e-4 for a membrane system, with temperatures within 7e-3 K and pressures within about 3 bar. See Accuracy and validation.

The difference that matters in practice is speed. On the development PC (RTX 5070 Ti, driver 610.88, CUDA 12.2 build) the GPU engine ran lysozyme at 1164 ns/day; the CPU engine reaches about 6 to 8 ns/day for a system of that size. The CPU engine is a reference implementation, not a production engine: a 100 ns production of a 100,000-atom complex is a matter of hours on the GPU and of weeks to months on the CPU.

How the engine is chosen

At every start the programs look for the NVIDIA driver, the GPUs and the FFT library, and the GPU engine is used when all are present and suitable. When it is not, the CPU engine is used and the program says why, in these places:

The Run MD page on a PC without a usable GPU. Look at the banner above the form: it names the reason (here the missing driver) and the remedy, and it stays there for as long as the program runs, so a slow run is never a surprise.
The Run MD page on a PC without a usable GPU. Look at the banner above the form: it names the reason (here the missing driver) and the remedy, and it stays there for as long as the program runs, so a slow run is never a surprise.
ReasonMessage (start)What to do
No driver"No NVIDIA graphics driver was found on this PC ..."Install the current driver if the PC has an NVIDIA GPU.
Driver too old"The NVIDIA driver of this PC (version X, supports CUDA up to N) is too old for the GPU engine. ..."Update the driver to 527.41 or newer and start the program again.
No CUDA device"The NVIDIA driver (...) reports no CUDA device ..."Check the device manager. If CUDA_VISIBLE_DEVICES is set, the message says it hides all devices; remove it.
GPU too old"GPU 0 (<name>) cannot run the GPU engine: <reason>. ..."Cards below compute capability 5.0 use the CPU engine with this build.
FFT library missing"The CUDA FFT library cufft64_11.dll was not found next to the program. ..."Restore bin from the package or install again.
Not enough GPU memory"... the GPU does not have enough free memory for this system. ..."Close other programs that use the GPU, or simulate a smaller system.
Start failure"The GPU engine could not be started on <device> (<what>)"Update the driver; run the self test.

Forcing an engine

SettingEffectWhen to use it
Project key engine.backend"auto" (default), "cpu" or "gpu"."cpu" for a reference run; "gpu" to be sure a long job never runs slowly by accident.
mdsim mdrun -nb gpuA run that cannot use the GPU stops with an error instead of falling back.Scripts on a GPU workstation.
MDS_GPU_NO_CPU_FALLBACK=1The same for a failing start of the GPU engine.Unattended queues where a slow run would be worse than a stopped one.

Several GPUs and the choice of device

With several GPUs, the first suitable device in the CUDA order (fastest first) that has enough free memory is used. Concurrent replicas and queued jobs of one process are spread over the suitable devices. One run never uses more than one GPU. The window has no device control; to pin a device:

SettingEffect
MDS_GPU_ID=NPins device N, including 0. A number beyond the devices present is reported: "MDS_GPU_ID=N: this PC has M CUDA device(s), numbered from 0."
-gpu_id N (command line) or engine.gpu_id (project)Selects device N for N > 0; 0 and -1 mean automatic.
CUDA_VISIBLE_DEVICESHides devices from the program, as for any CUDA application.

On a laptop with integrated and NVIDIA graphics, only the NVIDIA GPU is a CUDA device, so the simulation runs there; the viewer draws with the adapter Windows assigns to the application.

GPU memory

Before allocating anything, the engine estimates the memory a system needs, about 400 MiB plus 2000 bytes per atom plus 48 bytes per PME grid point, and compares it with the free memory of the device. If it does not fit, the CPU engine is used. The estimate is deliberately generous: measured use was 0.2 to 0.3 GB for systems of 6,000 to 89,000 atoms, so any current card holds a typical protein-ligand complex.

Checkpoints and bit-for-bit continuation

Checkpoints are written in Geinium's own format with a checksum, every 5 minutes (project key engine.checkpoint_minutes) and at the end of each MD stage. Each is written to a temporary file and renamed, so a crash during writing cannot damage it, and the previous checkpoint is kept as <name>_prev.cpt and used if the newest is unreadable. "Pause" ends a stage at the next pair-search step, after writing a checkpoint.

EngineContinuation after a stop or a crash
GPU engineByte-identical to an uninterrupted run, after a clean stop and after a crash.
CPU engine, 1 threadByte-identical.
CPU engine, several threadsIn the tests the trajectory stayed identical, but the energy file can differ in the last bits, because the order of summation over threads varies.

A checkpoint is refused when it does not match the run: another number of atoms, another integrator or another time step. Energy minimisation has no checkpoint and starts again if interrupted.

What the GPU engine does not run

The protocols of Run MD, the Batch queue and the System Builder use only what the GPU engine supports, including the restraints of the staged membrane equilibration. A run input made by hand stops with a clear message if it asks for any of the following:

Testing the GPU of a PC

mdsim gpuinfo -selftest runs a built-in system of 4,779 atoms (27 ethanol molecules in 1512 waters, PME, SETTLE and LINCS) for 20 steps, twice on every usable GPU and once on the CPU engine, and compares the results with the tolerances of the test suite: forces within 5e-5 relative RMS, every energy term within 5e-6 at the first step and 2e-4 over the run, and the two GPU runs identical bit for bit. It takes about two seconds and ends with "RESULT: PASSED", or with "RESULT: FAILED" and exit code 1. On the development card the forces agreed to 8.7e-7 to 9.1e-7.

Run the self test once on every new PC and after every driver update. The GPU engine was executed on one card only (compute capability 12.0); the machine code for compute capability 6.1, 7.5, 8.6 and 8.9 was compiled but never run. On a GTX 10xx or RTX 20xx to 40xx card, the self test is the evidence that the engine computes correctly there.

Speed

Measured on the development PC with mdsim bench -nb gpu, 20,000 timed steps of the complete NPT protocol, median of three runs, GPU otherwise idle:

System2 fs (default)4 fs, hydrogen-mass repartitioning
Lysozyme, 33,752 atoms1164.4 ns/day2134.9 ns/day
A 67,156-atom protein-ligand system674.9 ns/day1246.4 ns/day

Each start of the GPU engine costs about 1.4 s outside the step loop, so very short runs reach less. Other cards will be faster or slower; mdsim bench measures a given system on a given PC. Details, multiple time stepping and concurrent runs are in Performance.