Introduction to MPI

Aveuh

326K views

GitHub

Open Source Your Knowledge, Become a Contributor

Technology knowledge has to be shared and made accessible for free. Join the movement.

Create Content

Previous: Point-to-point communications - conclusion Next: Broadcasting

Collective communications

We have learnt how to send a message from a specific process to another specific process. But sometimes, the applications require more global solutions. Let's take an obvious example and imagine we have ten different processes. Process 0 has an information to dispatch to every other processes. With what we just learnt, the way we would do this would be something like :

Collective example 1

#include <iostream>
#include <mpi.h>
int main(int argc, char **argv) {
  MPI_Init(&argc, &argv);
  
  int rank, size;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  // Initialising the data
  int buffer[5];
  if (rank==0) {
    // Storing some values in the buffer
    for (int i=0; i < 5; ++i)
      buffer[i] = i*i;
    
    for (int id_dest=1; id_dest < size; ++id_dest)
      MPI_Send(buffer, 5, MPI_INT, id_dest, 0, MPI_COMM_WORLD);
  }
  else {
    MPI_Recv(buffer, 5, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
  }
  std::cout << "Process #" << rank << "; Buffer = (";
  for (int i=0; i < 5; ++i)
    std::cout << buffer[i] << (i < 4 ? " " : "");
  std::cout << ")" << std::endl;
  
  MPI_Finalize();
}

If you run this code, you will see that the buffer has been dispatched to every process.

Now this might seem acceptable if you don't know about collective communications but there is a proper, easier and more straightforward way of doing this :

Collective example 2

#include <iostream>
#include <mpi.h>
int main(int argc, char **argv) {
  MPI_Init(&argc, &argv);
  
  int rank, size;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  // Initialising the data
  int buffer[5];
  if (rank==0) {
    // Storing some values in the buffer
    for (int i=0; i < 5; ++i)
      buffer[i] = i*i;
  }
  MPI_Bcast(buffer, 5, MPI_INT, 0, MPI_COMM_WORLD);
  std::cout << "Process #" << rank << "; Buffer = (";
  for (int i=0; i < 5; ++i)
    std::cout << buffer[i] << (i < 4 ? " " : "");
  std::cout << ")" << std::endl;
  
  MPI_Finalize();
}

This does the exact same thing, but as you can see, there is only one simpler call to a MPI function.

Types of collective communications

Collective communications allow us to exchange more easily information across all processes (of a communicator). But collective communications are not only useful for this, there are different types of collective communications suited for very different objectives. Let's review the ones we are going to see in this tutorial :

Broadcast : One process sends a message to every other process
Reduction : One process gets data from all the other processes and applies an operation on it (sum, minimum, maximum, etc.
Scatter : A single process partitions the data to send pieces to every other process
Gather : A single process assembles the data from different process in a buffer

Blocking or non-blocking ?

As for p2p communications, all globals come in blocking and non-blocking flavours. We will be giving examples only for the blocking versions, but you just need to add the I to switch to non-blocking mode (eg MPI_Bcast will become MPI_Ibcast). As for p2p communications, non-blocking globals require the use of MPI_Wait and MPI_Test to be completed correctly.

In the next lessons, we are going to see more deeply each of these types of collective communications (apart from All to all communications) and we will apply them to computational problems.

Open Source Your Knowledge: become a Contributor and help others learn. Create New Content

Open Source Your Knowledge, Become a Contributor

14/30 Introduction to collective communications

Collective communications

Types of collective communications

Blocking or non-blocking ?

PYTHON: BEGINNER QUIZ (10 Questions)

PYTHON STRING METHODS: center(), rjust() and ljust().

Python from Zero to Hero

Leetcode from zero to hero