Introduction to MPI

Custom types

As you might have noticed, all datatypes in MPI communications are atomic types : an element correspond to one singular value. Moreover, every sommunication force you to use a contiguous buffer with the same datatype. Sometimes, it might be more interesting to give additional information and meaning to the communications by creating higher-level structures. MPI allows us to do that in the form of derived or custom datatypes. To make our point, let's take a simple example :

Let's consider a system with $$ N$$ processes where all processes are charged with generating data while process 0 centralizes and stores the data. The data generated by the processes corresponds to this struct :

struct CustomData {
  int n_values;
  double dbl_values[10];
};

Every process generates $$ M$$ of these custom structures, and then send them to process 0. What we want here is a simple gather on process 0 of all the values, but we are limited at the moment with MPI and cannot do that in a simple way. If we wanted to send this kind of data structure with the knowledge we currently have, we would do it this way :

As you can see from this very naive version, everything looks a lot more complicated than it should be. First we have to separate the values from every process into two tables, one for integer values, one for double values. Also note how the indexing part starts to become confusing with linear indexing on the double table. Then we have to gather everything in two passes and finally unpack everything in the final structure.

This problem could be solved in a simpler way using derived datatypes. A datatype can be defined easily by specifying a sequence of couples. Each couple represent a block : (type, displacement). The type is one of the usual types used in MPI, while the displacement indicates the offset in bytes where this data block starts in memory. For instance, if we wanted to use a structure like this :

struct DataType {
  int int_val;
  char char_val;
  float float_val;
};

We could describe this, as : [(int, 0), (char, 4), (float, 5)]. As for the example above, well the description is a bit more complicated since we have 10 double each time, but the idea is the same. Now, there are multiple ways of creating datatypes in MPI. For instance, there is a dedicated way to repeat the same datatype multiple times. There is also a more complex way of creating datatypes by generating lists such as the one showed above. We are going to see the simpler version here and the complex in the following exercise.

Vectors

Of course the simplest form of custom datatype is the simple repetition of the same type of data. For instance, if we were handling points in a 3d reference frame, then we would like to manipulate a Point structure with three doubles in it. We can achieve this very simply using the MPI_Type_contiguous function. Its prototype is :

int MPI_Type_contiguous(int count, MPI_Datatype old_type, MPI_Datatype *new_type);

So if we want to create a vector datatype, we can easily do :

MPI_Datatype dt_point;
MPI_Type_contiguous(3, MPI_DOUBLE, &dt_point);

We are not entirely done here, we need to commit the datatype. The commit operation allows MPI to generate a formal description of the buffers you will be sending and receiving. This is a mandatory operation. If you don't commit but still use your new datatype in communications, you are most likely to end up with invalid datatype errors. You can commit by simply calling MPI_Type_commit.

MPI_Type_commit(&dt_point);

Then we can freely use this in communications:

Let's now move on to an exercise on custom datatypes.