Open Source Your Knowledge, Become a Contributor

Technology knowledge has to be shared and made accessible for free. Join the movement.

Create Content

Collective communications

We have learnt how to send a message from a specific process to another specific process. But sometimes, the applications require more global solutions. Let's take an obvious example and imagine we have ten different processes. Process 0 has an information to dispatch to every other processes. With what we just learnt, the way we would do this would be something like :

Collective example 1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#include <iostream>
#include <mpi.h>
int main(int argc, char **argv) {
MPI_Init(&argc, &argv);
int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
// Initialising the data
int buffer[5];
if (rank==0) {
// Storing some values in the buffer
for (int i=0; i < 5; ++i)
buffer[i] = i*i;
for (int id_dest=1; id_dest < size; ++id_dest)
MPI_Send(buffer, 5, MPI_INT, id_dest, 0, MPI_COMM_WORLD);
}
else {
MPI_Recv(buffer, 5, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}
std::cout << "Process #" << rank << "; Buffer = (";
for (int i=0; i < 5; ++i)
std::cout << buffer[i] << (i < 4 ? " " : "");
std::cout << ")" << std::endl;
MPI_Finalize();
}
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

If you run this code, you will see that the buffer has been dispatched to every process.

Now this might seem acceptable if you don't know about collective communications but there is a proper, easier and more straightforward way of doing this :

Collective example 2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#include <iostream>
#include <mpi.h>
int main(int argc, char **argv) {
MPI_Init(&argc, &argv);
int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
// Initialising the data
int buffer[5];
if (rank==0) {
// Storing some values in the buffer
for (int i=0; i < 5; ++i)
buffer[i] = i*i;
}
MPI_Bcast(buffer, 5, MPI_INT, 0, MPI_COMM_WORLD);
std::cout << "Process #" << rank << "; Buffer = (";
for (int i=0; i < 5; ++i)
std::cout << buffer[i] << (i < 4 ? " " : "");
std::cout << ")" << std::endl;
MPI_Finalize();
}
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

This does the exact same thing, but as you can see, there is only one simpler call to a MPI function.

Types of collective communications

Collective communications allow us to exchange more easily information across all processes (of a communicator). But collective communications are not only useful for this, there are different types of collective communications suited for very different objectives. Let's review the ones we are going to see in this tutorial :

  • Broadcast : One process sends a message to every other process
  • Reduction : One process gets data from all the other processes and applies an operation on it (sum, minimum, maximum, etc.
  • Scatter : A single process partitions the data to send pieces to every other process
  • Gather : A single process assembles the data from different process in a buffer

Blocking or non-blocking ?

As for p2p communications, all globals come in blocking and non-blocking flavours. We will be giving examples only for the blocking versions, but you just need to add the I to switch to non-blocking mode (eg MPI_Bcast will become MPI_Ibcast). As for p2p communications, non-blocking globals require the use of MPI_Wait and MPI_Test to be completed correctly.

In the next lessons, we are going to see more deeply each of these types of collective communications (apart from All to all communications) and we will apply them to computational problems.

Open Source Your Knowledge: become a Contributor and help others learn. Create New Content