With MPI disabled (-DUSE_MPI='none'), the fallback routine for ORBIT_MPI_Allreduce performs a memcpy over n*sizeof(void*) (n*8 bytes), which is fine most of the time since we're typically copying an array of double (8 bytes). Where it's less-fine is when we're copying int or float (4 bytes), so the copy will be larger than necessary. If using long double (16 bytes), then the copy will not be performed over the full data.
|
/** A C wrapper around MPI_Allreduce. */ |
|
int ORBIT_MPI_Allreduce(void* ar1, void* ar2, int n, MPI_Datatype data, MPI_Op op, MPI_Comm comm){ |
|
int res = 0; |
|
#if USE_MPI > 0 |
|
res = MPI_Allreduce(ar1, ar2, n, data, op, comm); |
|
#else |
|
memcpy(ar2, ar1, n*sizeof(ar1)); |
|
res = MPI_SUCCESS; |
|
#endif |
|
return res; |
|
} |
We could probably create a switch on MPI_Datatype as they are defined manually in orbit_mpi.hh when MPI is off to ensure the right size is passed to memcpy, .e.g.,
static size_t orbit_mpi_datatype_size(MPI_Datatype dtype) {
switch (dtype) {
case MPI_CHAR: return sizeof(char);
case MPI_UNSIGNED_CHAR: return sizeof(unsigned char);
case MPI_BYTE: return sizeof(unsigned char);
case MPI_SHORT: return sizeof(short);
case MPI_UNSIGNED_SHORT: return sizeof(unsigned short);
case MPI_INT: return sizeof(int);
case MPI_UNSIGNED: return sizeof(unsigned);
case MPI_LONG: return sizeof(long);
case MPI_UNSIGNED_LONG: return sizeof(unsigned long);
case MPI_FLOAT: return sizeof(float);
case MPI_DOUBLE: return sizeof(double);
case MPI_LONG_DOUBLE: return sizeof(long double);
case MPI_LONG_LONG_INT: return sizeof(long long);
default: return 0;
}
}
With MPI disabled (
-DUSE_MPI='none'), the fallback routine for ORBIT_MPI_Allreduce performs amemcpyovern*sizeof(void*)(n*8 bytes), which is fine most of the time since we're typically copying an array of double (8 bytes). Where it's less-fine is when we're copyingintorfloat(4 bytes), so the copy will be larger than necessary. If usinglong double(16 bytes), then the copy will not be performed over the full data.PyORBIT3/src/mpi/orbit_mpi.cc
Lines 554 to 564 in 51f0568
We could probably create a switch on
MPI_Datatypeas they are defined manually inorbit_mpi.hhwhen MPI is off to ensure the right size is passed to memcpy, .e.g.,