The simulation engine and the GPU
The GPU engine and the CPU reference engine, how one is chosen, how a device is picked, memory, checkpoints, what the GPU engine does not run, and how to test a PC.
Two engines
| Engine | Precision | Where it runs | Use |
|---|---|---|---|
| GPU engine | Mixed: single-precision coordinates and pair forces, 64-bit fixed-point accumulation of forces, double-precision energies and virial. | Everything on the device: pair search, non-bonded forces, PME, bonded forces, update, LINCS and SETTLE, temperature and pressure coupling, centre-of-mass removal. Each step is one replayed sequence of GPU work. | Every simulation on a PC with a usable NVIDIA GPU. |
| CPU engine | Double precision throughout. | The processor, on several threads (at most 8 for jobs from the window). | The reference implementation: the fall-back without a GPU, and always for energy minimisation. |
Both engines implement the same algorithms and give the same results within the precision of each. On lysozyme (33,752 atoms) the forces of the GPU engine differed from those of the CPU engine by 1.3e-6 relative RMS and the potential energy by 2.1e-9 relative. Over a 20-step run the largest deviation of any energy term was 2.5e-5 for lysozyme and 1.0e-4 for a membrane system, with temperatures within 7e-3 K and pressures within about 3 bar. See Accuracy and validation.
The difference that matters in practice is speed. On the development PC (RTX 5070 Ti, driver 610.88, CUDA 12.2 build) the GPU engine ran lysozyme at 1164 ns/day; the CPU engine reaches about 6 to 8 ns/day for a system of that size. The CPU engine is a reference implementation, not a production engine: a 100 ns production of a 100,000-atom complex is a matter of hours on the GPU and of weeks to months on the CPU.
How the engine is chosen
At every start the programs look for the NVIDIA driver, the GPUs and the FFT library, and the GPU engine is used when all are present and suitable. When it is not, the CPU engine is used and the program says why, in these places:
- A banner at the top of the Run MD page and a "System" card on the Home page, with the reason and what to do.
- The status bar: "No hardware acceleration", with the explanation as its tooltip.
- The log of the run: "The GPU engine is not used for this run (<reason>). This run uses the CPU engine." and "Simulations run on the CPU engine: the results are the same, the run takes much longer."
- On the Simulate page, a line "<stage> runs on the CPU: <reason>".
| Reason | Message (start) | What to do |
|---|---|---|
| No driver | "No NVIDIA graphics driver was found on this PC ..." | Install the current driver if the PC has an NVIDIA GPU. |
| Driver too old | "The NVIDIA driver of this PC (version X, supports CUDA up to N) is too old for the GPU engine. ..." | Update the driver to 527.41 or newer and start the program again. |
| No CUDA device | "The NVIDIA driver (...) reports no CUDA device ..." | Check the device manager. If CUDA_VISIBLE_DEVICES is set, the message says it hides all devices; remove it. |
| GPU too old | "GPU 0 (<name>) cannot run the GPU engine: <reason>. ..." | Cards below compute capability 5.0 use the CPU engine with this build. |
| FFT library missing | "The CUDA FFT library cufft64_11.dll was not found next to the program. ..." | Restore bin from the package or install again. |
| Not enough GPU memory | "... the GPU does not have enough free memory for this system. ..." | Close other programs that use the GPU, or simulate a smaller system. |
| Start failure | "The GPU engine could not be started on <device> (<what>)" | Update the driver; run the self test. |
Forcing an engine
| Setting | Effect | When to use it |
|---|---|---|
Project key engine.backend | "auto" (default), "cpu" or "gpu". | "cpu" for a reference run; "gpu" to be sure a long job never runs slowly by accident. |
mdsim mdrun -nb gpu | A run that cannot use the GPU stops with an error instead of falling back. | Scripts on a GPU workstation. |
MDS_GPU_NO_CPU_FALLBACK=1 | The same for a failing start of the GPU engine. | Unattended queues where a slow run would be worse than a stopped one. |
Several GPUs and the choice of device
With several GPUs, the first suitable device in the CUDA order (fastest first) that has enough free memory is used. Concurrent replicas and queued jobs of one process are spread over the suitable devices. One run never uses more than one GPU. The window has no device control; to pin a device:
| Setting | Effect |
|---|---|
MDS_GPU_ID=N | Pins device N, including 0. A number beyond the devices present is reported: "MDS_GPU_ID=N: this PC has M CUDA device(s), numbered from 0." |
-gpu_id N (command line) or engine.gpu_id (project) | Selects device N for N > 0; 0 and -1 mean automatic. |
CUDA_VISIBLE_DEVICES | Hides devices from the program, as for any CUDA application. |
On a laptop with integrated and NVIDIA graphics, only the NVIDIA GPU is a CUDA device, so the simulation runs there; the viewer draws with the adapter Windows assigns to the application.
GPU memory
Before allocating anything, the engine estimates the memory a system needs, about 400 MiB plus 2000 bytes per atom plus 48 bytes per PME grid point, and compares it with the free memory of the device. If it does not fit, the CPU engine is used. The estimate is deliberately generous: measured use was 0.2 to 0.3 GB for systems of 6,000 to 89,000 atoms, so any current card holds a typical protein-ligand complex.
Checkpoints and bit-for-bit continuation
Checkpoints are written in Geinium's own format with a checksum, every 5 minutes (project key engine.checkpoint_minutes) and at the end of each MD stage. Each is written to a temporary file and renamed, so a crash during writing cannot damage it, and the previous checkpoint is kept as <name>_prev.cpt and used if the newest is unreadable. "Pause" ends a stage at the next pair-search step, after writing a checkpoint.
| Engine | Continuation after a stop or a crash |
|---|---|
| GPU engine | Byte-identical to an uninterrupted run, after a clean stop and after a crash. |
| CPU engine, 1 thread | Byte-identical. |
| CPU engine, several threads | In the tests the trajectory stayed identical, but the energy file can differ in the last bits, because the order of summation over threads varies. |
A checkpoint is refused when it does not match the run: another number of atoms, another integrator or another time step. Energy minimisation has no checkpoint and starts again if interrupted.
What the GPU engine does not run
The protocols of Run MD, the Batch queue and the System Builder use only what the GPU engine supports, including the restraints of the staged membrane equilibration. A run input made by hand stops with a clear message if it asks for any of the following:
- integrators other than leap-frog (minimisation runs on the CPU engine);
- free-energy perturbation;
- LJ-PME, virtual sites, freeze or acceleration groups, walls, pulling, distance restraints;
- SHAKE constraints;
- a PME interpolation order other than 4, or an Ewald tolerance below 1e-6;
- more than 16 temperature-coupling groups;
- multiple time stepping other than factor 2 for the long-range non-bonded forces.
Testing the GPU of a PC
mdsim gpuinfo -selftest runs a built-in system of 4,779 atoms (27 ethanol molecules in 1512 waters, PME, SETTLE and LINCS) for 20 steps, twice on every usable GPU and once on the CPU engine, and compares the results with the tolerances of the test suite: forces within 5e-5 relative RMS, every energy term within 5e-6 at the first step and 2e-4 over the run, and the two GPU runs identical bit for bit. It takes about two seconds and ends with "RESULT: PASSED", or with "RESULT: FAILED" and exit code 1. On the development card the forces agreed to 8.7e-7 to 9.1e-7.
Speed
Measured on the development PC with mdsim bench -nb gpu, 20,000 timed steps of the complete NPT protocol, median of three runs, GPU otherwise idle:
| System | 2 fs (default) | 4 fs, hydrogen-mass repartitioning |
|---|---|---|
| Lysozyme, 33,752 atoms | 1164.4 ns/day | 2134.9 ns/day |
| A 67,156-atom protein-ligand system | 674.9 ns/day | 1246.4 ns/day |
Each start of the GPU engine costs about 1.4 s outside the step loop, so very short runs reach less. Other cards will be faster or slower; mdsim bench measures a given system on a given PC. Details, multiple time stepping and concurrent runs are in Performance.