Have fun with MPI in C

spagnuolocarmine

135.7K views

GitHub

Open Source Your Knowledge, Become a Contributor

Technology knowledge has to be shared and made accessible for free. Join the movement.

Create Content

Previous: Blocking Communication Next: Non-Blocking Communication

Communication Modes

The send call described is blocking: it does not return until the message data and envelope have been safely stored away so that the sender is free to modify thesend buffer. The message might be copied directly into the matching receive buffer, or it might be copied into a temporary system buffer. Message buffering decouples the send and receive operations. A blocking send can complete as soon as the message was buffered, even if no matching receive has been executed bythe receiver. On the other hand, message buffering can be expensive, as it entails additionalmemory-to-memory copying, and it requires the allocation of memory for buffering.

MPI offers the choice of several communication modes that allow one to control the choice of the communication protocol. The send call described uses the standard communication mode. In this mode, it is up to MPI to decide whether outgoing messages will be buffered. MPI may buffer outgoing messages. In such a case, the send call may complete before a matching receive is invoked. On the other hand, buffer space may be unavailable, or MPI may choose not to buffer outgoing messages, for performance reasons. In this case, the send call will not complete until a matching receive has been posted, and the data has been moved to the receiver.

Thus, a send in standard mode can be started whether or not a matching receive hasbeen posted. It may complete before a matching receive is posted. The standard mode send is non-local: successful completion of the send operation may depend on the occurrence of a matching receive.

The communication mode is indicated by a one letter prefix:

B for buffered;
S for synchronous;
R for ready.

Buffered Mode

A buffered mode send operation can be started whether or not a matching receivehas been posted. It may complete before a matching receive is posted. However, unlike the standard send, this operation islocal, and its completion does not depend on the occurrence of a matching receive. Thus, if a send is executed and no matching receive is posted, then MPI must buffer the outgoing message, so as to allow the send call to complete. An error willoccur if there is insufficient buffer space. The amount of available buffer space is controlled by the user. Buffer allocation by the user may be required for the bufferedmode to be effective.

buffered_send

MPI_BSEND (buf, count, datatype, dest, tag, comm)

IN buf, initial address of send buffer (choice)
IN count, number of elements in send buffer (non-negative integer)
IN datatype, datatype of each send buffer element (handle)
IN dest, rank of destination (integer)
IN tag, message tag (integer)
IN comm, communicator (handle)

C version

int MPI_Bsend(const void* buf, int count, MPI_Datatype datatype, int dest,int tag, MPI_Comm comm)

The following example uses 2 processes.

MPI_BSEND

#include "mpi.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(argc,argv) int argc; char *argv[];
{
    int numtasks, rank, dest, source, rc, count;  
    char *inmsg;
    char *outmsg = "Testing";
    MPI_Status Stat;
    int bufsize = strlen(outmsg) * sizeof(char);
    char *buf = malloc(bufsize); 
    inmsg = (char *) malloc(10 * sizeof(char));
    MPI_Init(&argc,&argv);
    MPI_Comm_size(MPI_COMM_WORLD, &numtasks);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    if (rank == 0) {
        MPI_Buffer_attach( buf, bufsize ); 
        int time = MPI_Wtime();
        MPI_Bsend(&outmsg, strlen(outmsg), MPI_CHAR, 1, 1, MPI_COMM_WORLD);
        float etime =  MPI_Wtime() - time;
        printf("Task %d: buffered send (1) buffered , process will block on detach. Time: %1.2f\n",rank,etime);
        MPI_Buffer_detach( &buf, &bufsize ); 
        time = MPI_Wtime();
        MPI_Send(&outmsg, strlen(outmsg), MPI_CHAR, 1, 1, MPI_COMM_WORLD);
        etime =  MPI_Wtime() - time;
        printf("Task %d: buffered send (2), process may will block here. Time: %1.2f\n",rank, etime);
        
    } else {
        MPI_Recv(&inmsg, strlen(outmsg), MPI_CHAR, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &Stat);
        rc = MPI_Get_count(&Stat, MPI_CHAR, &count);
        printf("Task %d: Received %d char(s) from task %d with tag %d \n",
           rank, count, Stat.MPI_SOURCE, Stat.MPI_TAG);
        MPI_Recv(&inmsg, strlen(outmsg), MPI_CHAR, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &Stat);
        rc = MPI_Get_count(&Stat, MPI_CHAR, &count);
        printf("Task %d: Received %d char(s) from task %d with tag %d \n",
           rank, count, Stat.MPI_SOURCE, Stat.MPI_TAG);
    }
    MPI_Finalize();
}

Synchronous Mode

A send that uses the synchronous mode can be started whether or not a matching receive was posted. However, the send will complete successfully only if a matching receive is posted, and the receive operation has started to receive the message sent by the synchronous send. Thus, the completion of a synchronous send not only indicates that the send buffer can be reused, but it also indicates that the receiver has reached a certain point in its execution, namely that it has started executing the matching receive. If both sends and receives are blocking operations then the use of the synchronous mode provides synchronous communication semantics: a communication does not complete at either end before both processes rendezvous at the communication. A send executed in this mode is non-local.

synch_send

MPI_SSEND (buf, count, datatype, dest, tag, comm)

IN buf, initial address of send buffer (choice)
IN count, number of elements in send buffer (non-negative integer)
IN datatype, datatype of each send buffer element (handle)
IN dest, rank of destination (integer)
IN tag, message tag (integer)
IN comm, communicator (handle) å C version

int MPI_Ssend(const void* buf, int count, MPI_Datatype datatype, int dest,int tag, MPI_Comm comm)

The following example uses 2 processes.

MPI_Ssend

#include "mpi.h"
#include <stdio.h>
 
int main(int argc, char *argv[])
{
    int rank, size, i;
    int buffer[10];
    MPI_Status status;
 
    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    if (size < 2)
    {
        printf("Please run with two processes.\n");fflush(stdout);
        MPI_Finalize();
        return 0;
    }
    if (rank == 0)
    {
        for (i=0; i<10; i++)
            buffer[i] = i;
        MPI_Ssend(buffer, 10, MPI_INT, 1, 123, MPI_COMM_WORLD);
    }
    if (rank == 1)
    {
        for (i=0; i<10; i++)
            buffer[i] = -1;
        MPI_Recv(buffer, 10, MPI_INT, 0, 123, MPI_COMM_WORLD, &status);
        for (i=0; i<10; i++)
        {
            if (buffer[i] != i)
                printf("Error: buffer[%d] = %d but is expected to be %d\n", i, buffer[i], i);
        }
        fflush(stdout);
    }
    MPI_Finalize();
    return 0;
}

Ready Mode

A send that uses the ready communication mode may be started only if the matching receive is already posted. Otherwise, the operation is erroneous and its outcome is undefined. On some systems, this allows the removal of a hand-shake operation that is otherwise required and results in improved performance. The completion of the send operation does not depend on the status of a matching receive, and merely indicates that the send buffercan be reused. A send operation that uses the ready mode has the same semantics as astandard send operation, or a synchronous send operation; it is merely that the sender provides additional information to the system (namely that a matching receive is already posted), that can save some overhead. In a correct program, therefore, a ready send could be replaced by a standard send with no effect on the behavior of the program other than performance.

ready_send

MPI_RSEND (buf, count, datatype, dest, tag, comm)

IN buf, initial address of send buffer (choice)
IN count, number of elements in send buffer (non-negative integer)
IN datatype, datatype of each send buffer element (handle)
IN dest, rank of destination (integer)
IN tag, message tag (integer)
IN comm, communicator (handle)

C version

int MPI_Rsend(const void* buf, int count, MPI_Datatype datatype, int dest,int tag, MPI_Comm comm)

Notice that the following example uses the MPI_Irecv and MPI_Wait, which is a nonblocking receive and checking function, that will be described in the next chapter. Here you can consider that this kind of receive immediately returns.

The following example uses 2 processes.

MPI_Rsend

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
 
int main(int argc, char* argv[])
{
    MPI_Init(&argc, &argv);
 
   
    int size;
    MPI_Comm_size(MPI_COMM_WORLD, &size);
 
   
    enum role_ranks { SENDER, RECEIVER };
    int my_rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
    switch(my_rank)
    {
        case SENDER:
        {
            printf("MPI process %d hits the barrier to wait for the matching MPI_Recv to be posted.\n", my_rank);
            MPI_Barrier(MPI_COMM_WORLD);
            printf("The barrier unlocked, which means the MPI_Recv is already posted so the MPI_Rsend can be issued.\n");
 
            int buffer_sent = 12345;
            printf("MPI process %d sends value %d.\n", my_rank, buffer_sent);
            MPI_Rsend(&buffer_sent, 1, MPI_INT, RECEIVER, 0, MPI_COMM_WORLD);
            break;
        }
        case RECEIVER:
        {
            int received;
            MPI_Request request;
            MPI_Irecv(&received, 1, MPI_INT, SENDER, 0, MPI_COMM_WORLD, &request);
 
            printf("MPI process %d issued the MPI_Irecv, moved on and hit the barrier.\n", my_rank);
            MPI_Barrier(MPI_COMM_WORLD);
 
            MPI_Wait(&request, MPI_STATUS_IGNORE);
            printf("MPI process %d receives value %d.\n", my_rank, received);
            break;
        }
    }
 
    MPI_Finalize();
 
    return EXIT_SUCCESS;
}

Open Source Your Knowledge: become a Contributor and help others learn. Create New Content

Open Source Your Knowledge, Become a Contributor

8/22 Communication Modes

Communication Modes

Buffered Mode

Synchronous Mode

Ready Mode

La courbe du Dragon - S5

HEPL - Bloc 1, Q1 - Labos C - P Worontzoff

Programmation appliquée à l'électronique - Niveau II

Programação C