MPI_Pack_external_size is returning the wrong values in 64-bit applications

[ompiteam]

We have a program that tests for the size returned from MPI_Pack_external_size with the external32 data representation. It should return the same value for both 32-bit and 64-bit applications, but it is returning different values.

 burl-ct-v40z-0 65 =>mpicc ext32.c -o ext32
"ext32.c", line 105: warning: shift count negative or too big: << 32
 burl-ct-v40z-0 66 =>mpirun -np 2 ext32
First test passed
Second test passed
Third test passed
ext32: PASSED

 burl-ct-v40z-0 67 =>mpicc -xarch=amd64 ext32.c -o ext32_amd64
 burl-ct-v40z-0 68 =>mpirun -np 2 ext32_amd64 
First test passed
Second test failed. Got size of 80, expected 40
Third test failed. Got size of 6400, expected 3200
[burl-ct-v40z-0:13864] *** An error occurred in MPI_Pack_external
[burl-ct-v40z-0:13864] *** on communicator MPI_COMM_WORLD
[burl-ct-v40z-0:13864] *** MPI_ERR_TRUNCATE: message truncated
[burl-ct-v40z-0:13864] *** MPI_ERRORS_ARE_FATAL (goodbye)
 burl-ct-v40z-0 69 =>

[ompiteam]

Imported from trac issue 628. Created by rolfv on 2006-11-22T11:16:34, last modified: 2010-02-19T09:49:18

• rolfv attached ext32.c on 2006-11-22 11:17:17

[ompiteam]

Trac comment by rolfv on 2008-06-25 09:01:18:

I just retested this with the latest trunk and this still fails for us.

[ompiteam]

Trac comment by rolfv on 2008-09-09 14:39:08:

I just retested again with the trunk and with 1.3 and it still fails. It passes for 32-bits. Here are the 1.3 results (r19400).

 burl-ct-v440-0 47 =>mpirun -V
mpirun (Open MPI) 1.3r19400-ct8.0-b31c-r29

Report bugs to http://www.open-mpi.org/community/help/
 burl-ct-v440-0 48 =>

 burl-ct-v440-0 43 =>mpicc ext32.c -o ext32
"ext32.c", line 105: warning: shift count negative or too big: << 32
 burl-ct-v440-0 44 =>mpirun -np 2 ext32
First test passed
Second test passed
Third test passed
ext32: PASSED

 burl-ct-v440-0 46 =>mpirun -np 2 ext32_64
First test passed
Second test failed. Got size of 80, expected 40
Third test failed. Got size of 6400, expected 3200
[burl-ct-v440-0:04800] *** An error occurred in MPI_Pack_external
[burl-ct-v440-0:04800] *** on communicator MPI_COMM_WORLD
[burl-ct-v440-0:04800] *** MPI_ERR_TRUNCATE: message truncated
[burl-ct-v440-0:04800] *** MPI_ERRORS_ARE_FATAL (goodbye)
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 4800 on
node burl-ct-v440-0 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
 burl-ct-v440-0 47 =>

[ompiteam]

Trac comment by bosilca on 2008-09-17 10:05:17:

I did find some time to look at this one. And it's really funny ... I guess that your build of Open MPI do not support HETEROGENEOUS systems. Unfortunately, there are a lot of shortcuts when we know we are in a homogeneous environment, and one of those completely ignore the extern32 type (on 64 bits machines as they are different) when we create communicators ...

I'll try to figure out a nice solution, as even in heterogeneous systems the pack_extern32 should work as expected.

[ompiteam]

Trac comment by jsquyres on 2010-01-25 20:08:18:

Sun -- is this still happening (i.e., even after the changes in the DDT engine within the last few months)?

[ompiteam]

Trac comment by rolfv on 2010-01-26 16:59:55:

Yes, this still happens. I tried it on Solaris SPARC and a Linux machine using the Sun Studio compilers in both cases. Here is the Solaris SPARC failure.

 burl-ct-280r-0 55 =>mpicc -m64 ext32.c -o ext32_64
 burl-ct-280r-0 56 =>mpirun -V
mpirun (Open MPI) 1.4r22359-ct8.3-b08a-r13

Report bugs to http://www.open-mpi.org/community/help/
 burl-ct-280r-0 57 =>mpirun -np 2 ext32_64
First test passed
Second test failed. Got size of 80, expected 40
Third test failed. Got size of 6400, expected 3200
[burl-ct-280r-0:3402] *** An error occurred in MPI_Pack_external
[burl-ct-280r-0:3402] *** on communicator MPI_COMM_WORLD
[burl-ct-280r-0:3402] *** MPI_ERR_TRUNCATE: message truncated
[burl-ct-280r-0:3402] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 3402 on
node burl-ct-280r-0 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
 burl-ct-280r-0 58 =>

[ompiteam]

Trac comment by jsquyres on 2010-01-26 17:18:07:

George -- any hope of this getting fixed?

[ompiteam]

Trac comment by bosilca on 2010-02-08 15:43:17:

The root of the problem here is that if heterogeneous support is not enabled at build time, Open MPI does not support anything else than homogeneous environments. In this particular instance (i.e. 64 bits build and external representation), we are supposed to do a conversion from the local type into the external one. As heterogeneous support is disabled, we report the same size ...

[ompiteam]

Trac comment by jsquyres on 2010-02-08 15:59:54:

Ah, fair enough.

Should we just raise an MPI exception in this case (MPI_ERR_UNSUPPORTED or somesuch)?

[ompiteam]

Trac comment by rolfv on 2010-02-08 16:16:38:

I am not sure I follow all this. We configure our build with heterogeneous enabled.

 burl-ct-280r-0 58 =>ompi_info -all | grep Hetero
   Heterogeneous support: yes
 burl-ct-280r-0 60 =>mpirun -V
mpirun (Open MPI) 1.4r22359-ct8.3-b08a-r13

Report bugs to http://www.open-mpi.org/community/help/
 burl-ct-280r-0 61 =>

I also tried with the mca flag to force heterogeneous.

 burl-ct-280r-0 61 =>mpicc -m64 ext32.c -o ext32
 burl-ct-280r-0 62 =>mpirun -mca orte_hetero_apps 1 -np 1 ext32
First test passed
Second test failed. Got size of 80, expected 40
Third test failed. Got size of 6400, expected 3200
[burl-ct-280r-0:2958] *** An error occurred in MPI_Pack_external
[burl-ct-280r-0:2958] *** on communicator MPI_COMM_WORLD
[burl-ct-280r-0:2958] *** MPI_ERR_TRUNCATE: message truncated
[burl-ct-280r-0:2958] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 2958 on
node burl-ct-280r-0 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
 burl-ct-280r-0 63 =>

[ompiteam]

Trac comment by rusraink on 2010-02-19 09:49:18:

After inspecting and going through opal_convertor, this will most likely not make it to v1.5.

[bosilca]

few years old and no replicator.