Module mpp_mod

Contact:  V. Balaji
Change History: WebCVS Log
RCS Log: RCS Log


mpp_mod, is a set of simple calls to provide a uniform interface to different message-passing libraries. It currently can be implemented either in the SGI/Cray native SHMEM library or in the MPI standard. Other libraries (e.g MPI-2, Co-Array Fortran) can be incorporated as the need arises.

The data transfer between a processor and its own memory is based on load and store operations upon memory. Shared-memory systems (including distributed shared memory systems) have a single address space and any processor can acquire any data within the memory by load and store. The situation is different for distributed parallel systems. Specialized MPP systems such as the T3E can simulate shared-memory by direct data acquisition from remote memory. But if the parallel code is distributed across a cluster, or across the Net, messages must be sent and received using the protocols for long-distance communication, such as TCP/IP. This requires a ``handshaking'' between nodes of the distributed system. One can think of the two different methods as involving puts or gets (e.g the SHMEM library), or in the case of negotiated communication (e.g MPI), sends and recvs.

The difference between SHMEM and MPI is that SHMEM uses one-sided communication, which can have very low-latency high-bandwidth implementations on tightly coupled systems. MPI is a standard developed for distributed computing across loosely-coupled systems, and therefore incurs a software penalty for negotiating the communication. It is however an open industry standard whereas SHMEM is a proprietary interface. Besides, the puts or gets on which it is based cannot currently be implemented in a cluster environment (there are recent announcements from Compaq that occasion hope).

The message-passing requirements of climate and weather codes can be reduced to a fairly simple minimal set, which is easily implemented in any message-passing API. mpp_mod provides this API.

Features of mpp_mod include:

1) Simple, minimal API, with free access to underlying API for more complicated stuff.
2) Design toward typical use in climate/weather CFD codes.
3) Performance to be not significantly lower than any native API.

This module is used to develop higher-level calls for domain decomposition and parallel I/O.

Parallel computing is initially daunting, but it soon becomes second nature, much the way many of us can now write vector code without much effort. The key insight required while reading and writing parallel code is in arriving at a mental grasp of several independent parallel execution streams through the same code (the SPMD model). Each variable you examine may have different values for each stream, the processor ID being an obvious example. Subroutines and function calls are particularly subtle, since it is not always obvious from looking at a call what synchronization between execution streams it implies. An example of erroneous code would be a global barrier call (see mpp_sync below) placed within a code block that not all PEs will execute, e.g:

   if( pe.EQ.0 )call mpp_sync()
Here only PE 0 reaches the barrier, where it will wait indefinitely. While this is a particularly egregious example to illustrate the coding flaw, more subtle versions of the same are among the most common errors in parallel code.

It is therefore important to be conscious of the context of a subroutine or function call, and the implied synchronization. There are certain calls here (e.g mpp_declare_pelist, mpp_init, mpp_malloc, mpp_set_stack_size) which must be called by all PEs. There are others which must be called by a subset of PEs (here called a pelist) which must be called by all the PEs in the pelist (e.g mpp_max, mpp_sum, mpp_sync). Still others imply no synchronization at all. I will make every effort to highlight the context of each call in the MPP modules, so that the implicit synchronization is spelt out.

For performance it is necessary to keep synchronization as limited as the algorithm being implemented will allow. For instance, a single message between two PEs should only imply synchronization across the PEs in question. A global synchronization (or barrier) is likely to be slow, and is best avoided. But codes first parallelized on a Cray T3E tend to have many global syncs, as very fast barriers were implemented there in hardware.

Another reason to use pelists is to run a single program in MPMD mode, where different PE subsets work on different portions of the code. A typical example is to assign an ocean model and atmosphere model to different PE subsets, and couple them concurrently instead of running them serially. The MPP module provides the notion of a current pelist, which is set when a group of PEs branch off into a subset. Subsequent calls that omit the pelist optional argument (seen below in many of the individual calls) assume that the implied synchronization is across the current pelist. The calls mpp_root_pe and mpp_npes also return the values appropriate to the current pelist. The mpp_set_current_pelist call is provided to set the current pelist.




F90 is a strictly-typed language, and the syntax pass of the compiler requires matching of type, kind and rank (TKR). Most calls listed here use a generic type, shown here as MPP_TYPE_. This is resolved in the pre-processor stage to any of a variety of types. In general the MPP operations work on 4-byte and 8-byte variants of integer, real, complex, logical variables, of rank 0 to 5, leading to 48 specific module procedures under the same generic interface. Any of the variables below shown as MPP_TYPE_ is treated in this way.
Error handler.
Initialize mpp_mod.
Exit mpp_mod.
Symmetric memory allocation.
Allocate module internal workspace.
Reduction operations.
Reduction operation.
Basic message-passing call.
Parallel broadcasts.
Parallel checksums.




  1. mpp_error

    call mpp_error ( errortype, routine, errormsg )
    It is strongly recommended that all error exits pass through mpp_error to assure the program fails cleanly. An individual PE encountering a STOP statement, for instance, can cause the program to hang. The use of the STOP statement is strongly discouraged.

    Calling mpp_error with no arguments produces an immediate error exit, i.e:
        call mpp_error
        call mpp_error(FATAL)
    are equivalent.

    The argument order
        call mpp_error( routine, errormsg, errortype )
    is also provided to support legacy code. In this version of the call, none of the arguments may be omitted.

    The behaviour of mpp_error for a WARNING can be controlled with an additional call mpp_set_warn_level.
        call mpp_set_warn_level(ERROR)
    causes mpp_error to treat WARNING exactly like FATAL.
        call mpp_set_warn_level(WARNING)
    resets to the default behaviour described above.

    mpp_error also has an internal error state which maintains knowledge of whether a warning has been issued. This can be used at startup in a subroutine that checks if the model has been properly configured. You can generate a series of warnings using mpp_error, and then check at the end if any warnings has been issued using the function mpp_error_state(). If the value of this is WARNING, at least one warning has been issued, and the user can take appropriate action:

        if( ... )call mpp_error( WARNING, '...' )
        if( ... )call mpp_error( WARNING, '...' )
        if( ... )call mpp_error( WARNING, '...' )
        if( mpp_error_state().EQ.WARNING )call mpp_error( FATAL, '...' )

    errortype    One of NOTE, WARNING or FATAL (these definitions are acquired by use association). NOTE writes errormsg to STDOUT. WARNING writes errormsg to STDERR. FATAL writes errormsg to STDERR, and induces a clean error exit with a call stack traceback.

  2. mpp_init

    call mpp_init ( flags )
    Called to initialize the mpp_mod package. It is recommended that this call be the first executed line in your program. It sets the number of PEs assigned to this run (acquired from the command line, or through the environment variable NPES), and associates an ID number to each PE. These can be accessed by calling mpp_npes and mpp_pe.

    flags    flags can be set to MPP_VERBOSE to have mpp_mod keep you informed of what it's up to.

  3. mpp_exit

    call mpp_exit ()
    Called at the end of the run, or to re-initialize mpp_mod, should you require that for some odd reason.

    This call implies synchronization across all PEs.

  4. mpp_malloc

    call mpp_malloc ( ptr, newlen, len )
    This routine is used on SGI systems when mpp_mod is invoked in the SHMEM library. It ensures that dynamically allocated memory can be used with shmem_get and shmem_put. This is called symmetric allocation and is described in the intro_shmem man page. ptr is a Cray pointer (see the section on portability). The operation can be expensive (since it requires a global barrier). We therefore attempt to re-use existing allocation whenever possible. Therefore len and ptr must have the SAVE attribute in the calling routine, and retain the information about the last call to mpp_malloc. Additional memory is symmetrically allocated if and only if newlen exceeds len.

    This is never required on Cray PVP or MPP systems. While the T3E manpages do talk about symmetric allocation, mpp_mod is coded to remove this restriction.

    It is never required if mpp_mod is invoked in MPI.

    This call implies synchronization across all PEs.

    ptr    a cray pointer, points to a dummy argument in this routine.
    newlen    the required allocation length for the pointer ptr
    len    the current allocation (0 if unallocated).

  5. mpp_set_stack_size

    call mpp_set_stack_size (n)
    mpp_mod maintains a private internal array called mpp_stack for private workspace. This call sets the length, in words, of this array.

    The mpp_init call sets this workspace length to a default of 32768, and this call may be used if a longer workspace is needed.

    This call implies synchronization across all PEs.

    This workspace is symmetrically allocated, as required for efficient communication on SGI and Cray MPP systems. Since symmetric allocation must be performed by all PEs in a job, this call must also be called by all PEs, using the same value of n. Calling mpp_set_stack_size from a subset of PEs, or with unequal argument n, may cause the program to hang.

    If any MPP call using mpp_stack overflows the declared stack array, the program will abort with a message specifying the stack length that is required. Many users wonder why, if the required stack length can be computed, it cannot also be specified at that point. This cannot be automated because there is no way for the program to know if all PEs are present at that call, and with equal values of n. The program must be rerun by the user with the correct argument to mpp_set_stack_size, called at an appropriate point in the code where all PEs are known to be present.


  6. mpp_max

    call mpp_max ( a, pelist )
    Find the max of scalar a the PEs in pelist result is also automatically broadcast to all PEs

    a    real or integer, of 4-byte of 8-byte kind.
    pelist    If pelist is omitted, the context is assumed to be the current pelist. This call implies synchronization across the PEs in pelist, or the current pelist if pelist is absent.

  7. mpp_sum

    call mpp_sum ( a, length, pelist )
    MPP_TYPE_ corresponds to any 4-byte and 8-byte variant of integer, real, complex variables, of rank 0 or 1. A contiguous block from a multi-dimensional array may be passed by its starting address and its length, as in f77.

    Library reduction operators are not required or guaranteed to be bit-reproducible. In any case, changing the processor count changes the data layout, and thus very likely the order of operations. For bit-reproducible sums of distributed arrays, consider using the mpp_global_sum routine provided by the mpp_domains module.

    The bit_reproducible flag provided in earlier versions of this routine has been removed.

    If pelist is omitted, the context is assumed to be the current pelist. This call implies synchronization across the PEs in pelist, or the current pelist if pelist is absent.



  8. mpp_transmit

    call mpp_transmit ( put_data, put_len, put_pe, get_data, get_len, get_pe )
    MPP_TYPE_ corresponds to any 4-byte and 8-byte variant of integer, real, complex, logical variables, of rank 0 or 1. A contiguous block from a multi-dimensional array may be passed by its starting address and its length, as in f77.

    mpp_transmit is currently implemented as asynchronous outward transmission and synchronous inward transmission. This follows the behaviour of shmem_put and shmem_get. In MPI, it is implemented as mpi_isend and mpi_recv. For most applications, transmissions occur in pairs, and are here accomplished in a single call.

    The special PE designations NULL_PE, ANY_PE and ALL_PES are provided by use association.

    NULL_PE: is used to disable one of the pair of transmissions.
    ANY_PE: is used for unspecific remote destination. (Please note that put_pe=ANY_PE has no meaning in the MPI context, though it is available in the SHMEM invocation. If portability is a concern, it is best avoided).
    ALL_PES: is used for broadcast operations.

    It is recommended that mpp_broadcast be used for broadcasts.

    The following example illustrates the use of NULL_PE and ALL_PES:

        real, dimension(n) :: a
        if( pe.EQ.0 )then
            do p = 1,npes-1
               call mpp_transmit( a, n, p, a, n, NULL_PE )
            end do
            call mpp_transmit( a, n, NULL_PE, a, n, 0 )
        end if
        call mpp_transmit( a, n, ALL_PES, a, n, 0 )
    The do loop and the broadcast operation above are equivalent.

    Two overloaded calls mpp_send and mpp_recv have also been provided. mpp_send calls mpp_transmit with get_pe=NULL_PE. mpp_recv calls mpp_transmit with put_pe=NULL_PE. Thus the do loop above could be written more succinctly:

        if( pe.EQ.0 )then
            do p = 1,npes-1
               call mpp_send( a, n, p )
            end do
            call mpp_recv( a, n, 0 )
        end if

  9. mpp_broadcast

    call mpp_broadcast ( data, length, from_pe, pelist )
    The mpp_broadcast call has been added because the original syntax (using ALL_PES in mpp_transmit) did not support a broadcast across a pelist.

    MPP_TYPE_ corresponds to any 4-byte and 8-byte variant of integer, real, complex, logical variables, of rank 0 or 1. A contiguous block from a multi-dimensional array may be passed by its starting address and its length, as in f77.

    Global broadcasts through the ALL_PES argument to mpp_transmit are still provided for backward-compatibility.

    If pelist is omitted, the context is assumed to be the current pelist. from_pe must belong to the current pelist. This call implies synchronization across the PEs in pelist, or the current pelist if pelist is absent.



  10. mpp_chksum

    mpp_chksum ( var, pelist )
    mpp_chksum is a parallel checksum routine that returns an identical answer for the same array irrespective of how it has been partitioned across processors. LONG_KINDis the KIND parameter corresponding to long integers (see discussion on OS-dependent preprocessor directives) defined in the header file fms_platform.h. MPP_TYPE_ corresponds to any 4-byte and 8-byte variant of integer, real, complex, logical variables, of rank 0 to 5.

    Integer checksums on FP data use the F90 TRANSFER() intrinsic.

    The serial checksum module is superseded by this function, and is no longer being actively maintained. This provides identical results on a single-processor job, and to perform serial checksums on a single processor of a parallel job, you only need to use the optional pelist argument.
         use mpp_mod
         integer :: pe, chksum
         real :: a(:)
         pe = mpp_pe()
         chksum = mpp_chksum( a, (/pe/) )
    The additional functionality of mpp_chksum over serial checksums is to compute the checksum across the PEs in pelist. The answer is guaranteed to be the same for the same distributed array irrespective of how it has been partitioned.

    If pelist is omitted, the context is assumed to be the current pelist. This call implies synchronization across the PEs in pelist, or the current pelist if pelist is absent.