So We Agreed on Having Multiple MPIs… What About SHMEM?

Jobs may be run in parallel using either MPI (Message Passing Interface) library or the SHMEM (put/get) library for process synchronizations and communications. Which one is better?

Let us go back to basics. The MPI library is a standard message passing library for parallel applications. Using MPI, parallel processes cooperate to perform their task by passing messages to each other. MPI includes point-to-point message passing (send / receive) and collective operations between a user-defined group of processes.

The SHMEM library provides direct access (put and get calls) to the memory of remote processes. A message passing library, such as MPI, requires that the remote process issues a receive to complete the transmission of each message. On the other hand, the SHMEM library provides the initiating process with direct access to the target memory.

SHMEM provides the following routine:  put routines write data to another process, get routines read data from another process, collective routines distribute work across a set of processes, atomic routines perform an atomic fetch-and-operate, such as fetch-and-increment or swap, synchronization routines order the actions of processes, and reduction routines reduce an array to a scalar value by performing a cumulative operation on some or all of the array elements.

SHMEM routines can provide high performance by minimizing the overhead associated with data passing requests. By performing a direct memory-to-memory copy, SHMEM can takes less steps to perform an operation. However, additional synchronization steps are almost always required when using SHMEM. Assuming that the job can be executed on both options, which one is better?

As I see it, it is more a religious kind of question… some folks prefer MPI, some folks prefer SHMEM saying that MPI cannot scale and SHMEM is the way for the exascale… really? I like to use MPI, and I don’t think that MPI is inferior to SHMEM. I do prefer the hybrid option of OpenMP inside and MPI outside, but MPI works great for me…