How do I run OpenIFS in parallel configuration using several nodes?

AIFS, IFS and OpenIFS OpenIFS
, ,
1

We have received several questions about how OpenIFS 48r1 can be used across multiple nodes on an HPC system. It may be helpful to summarise this here:

First case:
You want to run the model interactively, i.e. you execute the oifs-run script directly from the command line in your terminal.
(Note: Unless your system is very powerful, this will only work for low grid resolutions and short test runs.)

  • First, edit the exp-config.h file in your experiment directory, and set OIFS_NPROC and OIFS_NTHREAD to the number of MPI tasks and OpenMP threads you wish to use.
  • Also in this file, set the LAUNCH variable to the command that starts the model binary for parallel runs on your system. This might be mpirun, aprun, or srun—your local computing support team can advise you on which is appropriate.
    You do not need to include the OpenIFS executable name in this variable—only the command and its required options, e.g.: LAUNCH="mpirun -np ${OIFS_NPROC}"
  • The oifs-run script will check whether the LAUNCH variable has been set. If it hasn’t, it will attempt to create it using its default launcher settings, found at line 349ff in the script.
  • A configuration for the hpc2020 machine at ECMWF is included in the script. You may add your own system’s configuration there if you prefer not to set the LAUNCH command directly in exp-config.h.

Second case:
You want to run the model as a batch job—this is the more common method for running OpenIFS.

  • You will use a job script that calls the oifs-run script. We provide an example tailored for the hpc2020 system at ECMWF: $OIFS_RUN_SCRIPT/run-oifs.ecmwf-hpc2020.job
  • The header lines in this script request the resources for your job. The example provided (for the SLURM batch scheduler used at ECMWF) requests 8 nodes by default.
    You may need to adjust these headers to match your own HPC system. Your local computing support can assist with this.
  • In the job script, you should specify the location of your oifs-config file.
  • You do not need to edit OIFS_NPROC and OIFS_NTHREAD in your experiment’s exp-config.h file, as the job script will override these values before calling oifs-run.
  • In your experiment’s exp-config.h file, set the LAUNCH variable to the basic command without any flags, e.g. "srun".

More information can be found in the OpenIFS 48r1 User Guide - OpenIFS - ECMWF Confluence Wiki.

Note: The description above applies to version openifs-48r1-20250214.