Alternate Run Modes:MPI and OpenMP Parallelization

From Gsshawiki
Jump to: navigation, search

In GSSHA, there are three parallelization modes supported. When running a simulation, a control mode and a parallelization mode are both selected; the default for most runs is a single-threaded (serial), run once mode. Apart from the single-thread or serial mode, GSSHATM can be run in OpenMP or MPI mode.


OpenMP is a parallelization paradigm that assumes you are running on a machine with multiple logical processors and a shared pool of memory. When running, OpenMP splits up some portions of the code to run in parallel on multiple threads, while others are run on a single thread or processing unit.

The following is a list of the portions of the code that have been parallelized using OpenMP.

  • Overland boundary conditions
  • Rain gage nearest neighbor algorithm (for Theissen polygon interpolation)
  • Rain gage inverse distance algorithm (for IDW interpolation)
  • Green and Ampt infiltration algorithm
  • Multi-layer Green and Ampt infiltration algorithm
  • Green and Ampt w/ Soil Moisture Redistribution algorithm
  • Evapotranspiration algorithm
  • NSM stream kinetics
  • NSM overland kinetics
  • NSM soil-overland interaction
  • Explicit, ADE, and ADE-PC overland flow algorithms
  • Some parts of the SCE routine

There are some significant portions of the code that have not been parallelized with OpenMP yet, such as groundwater, Richard's infiltration, and constituent transport.

The speed-up from using OpenMP will depend on various factors such as:

  • How big is the simulation (how many cells)
  • What portion of work is done by the parallelized codes v. the non-parallelized codes
  • How many threads (logical processors) are you using (versus how many actual processing units you have)

The one factor you have control over for OpenMP, if you are running in single-run mode, is the number of threads you want to dedicate to a particular process. For batch, SCE, and monte carlo methods it sets the thread count to the total number of logical processors you have running. Note that if you have a CPU with hyperthreading then each processor on the CPU appears as two logical processors. Thus a dual-core CPU with hyperthreading turned on would appear as four logical processors. There is some improvement from hyperthreading, especially if a lot of writing to output files is being done.

To set the thread count for single-run modes, use the following project file card.


where ## is the number of threads to set; must be greater than or equal to 1. If you run a version of GSSHATM that was not compiled with OpenMP, the NUM_THREADS project card will result in a warning that it was not compiled with OpenMP; GSSHATM will then continue to run in single-thread mode.


Unlike OpenMP, MPI mode currently is only useful for running multiple versions of a single simulation. This is best suited for monte carlo runs on supercomputers. To run in MPI mode, you must use an external program that handles the setup and communication processes, such as MPICH. Typically these program require that you specify the GSSHATM command line as part of the command line for the MPI program. The GSSHATM command line for running the various control modes is specified as part of the control mode description in the following sections.

GSSHA User's Manual

18 Alternate Run Modes
18.1     MPI and OpenMP Parallelization
18.2     Simulation Setup for Alternate Run Modes
18.3     Batch Mode Runs
18.4     Automated Calibration with Shuffled Complex Evolution
18.5     Monte Carlo Runs
18.6     ERDC Automated Model Calibration Software
   18.6.1     Efficient Local Search
   18.6.2     Multistart
   18.6.3     Trajectory Repulsion
   18.6.4     Effective and Efficient Stochastic Global Optimization