MPI run on local machine • NONMEM Documentation

MPI run on a local machine

NONMEM supports parallel computation through Message Passing Interface (MPI). As modern hardware advacement gifts destop computation with multicore processors, so that MPI runs can be executed locally to speed up computation, as opposite to dispatched remotely to a server. This article describes the steps for local parallel runs using the precompiled NONMEM library. User need to ensure the build toolchain has been installed.

To utilize the parallel feature one must have the MPI library available. Please consult vendor documentation such as by Microsoft MPI, OpenMPI, and MPICH. Note that NONMEM contains libraries for Microsoft MPI (MSMPI), so users need only install the MSMPI executable but not the SDK.

Build

To build a parallel NONMEM executable, follow the common "configure-build-install" process and use the relevant MPI options.

Create a build directory at, say /path/to/build,

1

  mkdir /path/to/build && cd /path/to/build

Assume NONMEM is located at /path/to/nonmem, to build a model at /path/to/model/model-name.ctl, in the build directory, do
1 2

# configure cmake /path/to/nonmem -B. -GNinja -Dmodel=/path/to/model/model-name.ctl -Dmpi=ON
Here option -Dmpi=ON requests for MPI builds. One can use additional cmake options to customize MPI build. For example, one can use -DMPI_HOME= to help cmake locate the MPI library:
1

cmake /path/to/nonmem -B. -GNinja -Dmodel=/path/to/model/model-name.ctl -Dmpi=ON -DMPI_HOME=/path/to/mpich-4.3.2
The above line enables MPI build using the library mpich-4.3.2.

For Windows users with Microsoft MPI (MSMPI), a special option mpi=MSMPI
1

cmake /path/to/nonmem -B. -GNinja -Dmodel=/path/to/model/model-name.ctl -Dmpi=MSMPI
enables MSMPI build. The MPI_HOME option is then ignored.

The other steps are the same as the sequential build:

1
2
3
4


  # build
  cmake --build . -j2	# -j2: use 2 threads to build faster
  # install
  cmake --install .

Run

To run the model in parallel, do

1
2
3
4
5


  cd /path/to/model/
  # Windows
  mpiexec.exe -n num_nodes model-name.exe model-name.ctl output -licfile=/path/to/license/nonmem.lic -parafile=LOCALNODE
  # Linux & MacOS
  ./mpiexec -n num_nodes model-name model-name.ctl output -licfile=/path/to/license/nonmem.lic -parafile=LOCALNODE

Here -parafile=LOCALNODE requests a "local node" MPI run, and num_nodes is the number of cores user needs to specify. The subjects in the model will be distributed to the specified cores and their contribution to OFV will be calculated in paralle. For example,

1

./mpiexec -n 4 model-name model-name.ctl output -licfile=/path/to/license/nonmem.lic -parafile=LOCALNODE

says that the population will be distributed to four cores, assuming there are more than four cores in the machine.

User can explore additional vendor-depedent options in MPI runs. For example, it is often desirable to use the --bind-to core option

1

  ./mpiexec -n 4 --bind-to core #...

so that num_nodes would indicate the number of hardware cores, as opposite to hardware threads (vCPUs).

Run using `nmf2`

User can simplify the workflow by using nonmem/util/nmf2.sh (Linux/MacOS) or nonmem/util/nmf2.bat (Windows) scripts. These are thin wrappers of the above cmake procedures. See more about the script here.

The nodes= option of the nmf2 script enables MPI build and run. It specifies the number of cores from your local machine to be used. Note that by default nmf2 uses Microsoft MPI (MSMPI) to build the executable. To use an alternative MPI library, specify it using -mpi_home= option. Thus

1

nmf2.bat /path/to/model.ctl /path/to/output.res -nodes=4

will build the model using MSMPI and run it with 4 cores; while

1

nmf2.bat /path/to/model.ctl /path/to/output.res -nodes=4 -mpi_home=/path/to/mpich-4.3.2

will build the model using specified MPI library mpich-4.3.2 and run it with 4 cores.

MPI run on a local machine

Build

Run

Run using nmf2

Run using `nmf2`