Support little-endian powerpc linux host

[nysal]

Power 8 systems for HPC use little endian mode by default. Support for this is available in libtool 2.4.3. For older versions of libtool we need to patch libtool.m4 to add support. The following patch is what needs to be applied:

diff --git a/libltdl/m4/libtool.m4 b/libltdl/m4/libtool.m4
index d812584..7aefebc 100644
--- a/libltdl/m4/libtool.m4
+++ b/libltdl/m4/libtool.m4
@@ -1268,7 +1268,7 @@ ia64-*-hpux*)
   rm -rf conftest*
   ;;

-x86_64-*kfreebsd*-gnu|x86_64-*linux*|ppc*-*linux*|powerpc*-*linux*| \
+x86_64-*kfreebsd*-gnu|x86_64-*linux*|powerpc*-*linux*| \
 s390*-*linux*|s390*-*tpf*|sparc*-*linux*)
   # Find out which ABI we are using.
   echo 'int i;' > conftest.$ac_ext
@@ -1282,7 +1282,10 @@ s390*-*linux*|s390*-*tpf*|sparc*-*linux*)
      x86_64-*linux*)
        LD="${LD-ld} -m elf_i386"
        ;;
-     ppc64-*linux*|powerpc64-*linux*)
+     powerpc64le-*linux*)
+       LD="${LD-ld} -m elf32lppclinux"
+       ;;
+     powerpc64-*linux*)
        LD="${LD-ld} -m elf32ppclinux"
        ;;
      s390x-*linux*)
@@ -1301,7 +1304,10 @@ s390*-*linux*|s390*-*tpf*|sparc*-*linux*)
      x86_64-*linux*)
        LD="${LD-ld} -m elf_x86_64"
        ;;
-     ppc*-*linux*|powerpc*-*linux*)
+     powerpcle-*linux*)
+       LD="${LD-ld} -m elf64lppc"
+       ;;
+     powerpc-*linux*)
        LD="${LD-ld} -m elf64ppc"
        ;;
      s390*-*linux*|s390*-*tpf*)

For mpich we patch libtool.m4 and rebuild configure (http://git.mpich.org/mpich.git/commitdiff/b0a75d1660b89e8df453153c1c3b96d36fdb0f6b?hp=2e0747aac347153bdf4e7bf5efe638a455904bce). ompi seems to patch configure directly, for adding any autotool related patches. I'm open to suggestions on how this can be done cleanly.

[jsquyres]

I've been a little slow to answer this one, because the Right answer might well be to finish #311 and make the minimum requirement for 1.9 OMPI tarballs be Libtool 2.4.3 (or probably 2.4.4, since that's the most recent version).

@nysal would that be a better solution?

[nysal]

Yes, this might be a better approach. I hacked up a patch in the meanwhile and it didnt look pretty. FYI, Libtool seems to have raced ahead to version 2.4.6 as of today. 1.8.x series doesn't compile on powerpcle. I could ask folks to run autogen again in case they need to compile 1.8.x on powerpcle.

[jsquyres]

We weren't planning to bring the #311 solution to the v1.8 branch (it's going to be a large change). Something like your patch for the v1.8 branch may be the best solution there.

[jsquyres]

FYI: RFC to update master to later Autotools: http://www.open-mpi.org/community/lists/devel/2015/03/17177.php

[nysal]

Thanks Jeff. For 1.8.x series, I'll work on a patch this week.

[jsquyres]

@nysal The v1.8 series is effectively done. Do you want to make a patch for v1.10?

[gpaulsen]

Talked to Nysal, and he said:

ok, so the issue is that the 1.10 tarballs for ompi don't work on ppc64le. The reason being that 1.10 tarballs were generated using libtool 2.4.2. The earliest version of libtool that supports ppc64le is 2.4.3. ompi does not change libtool versions in a stable release series.
It works if you manually run autogen.pl after extracting the tarball.
The way to work around the problem is to patch configure with backports from libtool 2.4.3. I have provided the patch in the issue #396
We just need to modify autogen.pl to patch configure with the patch specified in #396
And verify it patched all the configure scripts. ompi emebds a few other projects - romio, libevent etc. We need to make sure each one of them gets patched.

[gpaulsen]

My question is, is it possible to update to libtool 2.4.3 for 1.10.1? Or is backporting the fix out of libtool 2.4.3 to 2.4.2 the more desirable approach?

[jsquyres]

We try to stay with the same version of the GNU Autotools for given release series -- just for stability.

So if you can add that patch that @nysal specified to the autogen.pl patching area, that would be preferable.

[gpaulsen]

Okay, I'll discuss with @Zhiming-Wang tonight to see if he's able to apply this fix. If not, @nysal said he could find some time.

The expectation is that this fix, will also fix Issue #922. I'm closing #922 as a dup of this.

[Zhiming-Wang]

Yes, issue #992 is a duplicate of this. Thanks @gpaulsen .
Today, I worked with PowerLinux team and make progress.
There is an auto-fix script "/usr/lib/rpm/redhat/libtool-handle-ppc64le.sh" at RHEL 7. The content of script is very similar as @nysal fix. So the issue was disappeared at Power8 LE platform when rerunning "./autogen.pl".
The fix is for libtool, not Open MPI. Will it be patched at Open MPI's packaging machine?

If a PPC LE user, he/she just need to rerun "./autogen.pl". Or manually run "libtool-handle-ppc64le.sh" at top directory of Open MPI. The fix is not needed.
If not PPC LE user, the issue will not appear.

[Zhiming-Wang]

 wangzm@ppcrhel7c-651: cat libtool-handle-ppc64le.sh
#!/bin/bash

# Date: April 2, 2014
# by Aldy Hernandez

# Attempt to fix any "-m elf64ppc" linker checks in configure and
# libtool.m4 files.
#
# This script is meant to run silently as part of rpm's %configure
# macro.  It either fixes the problem, or we silently ignore it, in
# which case it is up to the package maintainer to add support for
# ppc64le.

# Our two attempts at fixing the problem.
PATCH1=/tmp/$$.patch1
PATCH2=/tmp/$$.patch2

cleanup() {
    rm -f $PATCH1 $PATCH2
}

trap cleanup 0 1 2 3 4 5 6 7 8 9 11 13 14 15

# There are two variants in RHEL7 so far.  The first version, handled
# with $PATCH1, currently handles all but 3 packages.  The $PATCH2
# version handles the remnant.
#
cat > $PATCH1 <<EOF
--- configure.orig      2014-03-18 15:56:15.575070238 -0500
+++ configure   2014-03-18 16:05:50.877861163 -0500
@@ -7714,6 +7714,9 @@
          x86_64-*linux*)
            LD="\${LD-ld} -m elf_i386"
            ;;
+         ppc64le-*linux*|powerpc64le-*linux*)
+           LD="\${LD-ld} -m elf32lppclinux"
+           ;;
          ppc64-*linux*|powerpc64-*linux*)
            LD="\${LD-ld} -m elf32ppclinux"
            ;;
@@ -7733,6 +7736,9 @@
          x86_64-*linux*)
            LD="\${LD-ld} -m elf_x86_64"
            ;;
+         ppc*le-*linux*|powerpc*le-*linux*)
+           LD="\${LD-ld} -m elf64lppc"
+           ;;
          ppc*-*linux*|powerpc*-*linux*)
            LD="\${LD-ld} -m elf64ppc"
            ;;
EOF

cat > $PATCH2 <<EOF
--- configure.orig      2014-03-18 16:35:28.942799967 -0500
+++ configure   2014-03-18 16:34:35.608519090 -0500
@@ -3798,6 +3798,9 @@
         x86_64-*linux*)
           LD="\${LD-ld} -m elf_i386"
           ;;
+        ppc64le-*linux*)
+          LD="\${LD-ld} -m elf32lppclinux"
+          ;;
         ppc64-*linux*)
           LD="\${LD-ld} -m elf32ppclinux"
           ;;
@@ -3814,6 +3817,9 @@
         x86_64-*linux*)
           LD="\${LD-ld} -m elf_x86_64"
           ;;
+        ppc*le-*linux*|powerpc*le-*linux*)
+          LD="\${LD-ld} -m elf64lppc"
+          ;;
         ppc*-*linux*|powerpc*-*linux*)
           LD="\${LD-ld} -m elf64ppc"
           ;;
EOF


FILES=`find . -name configure -o -name libtool.m4`
for f in $FILES; do
    # Filter out candidates that already handle ppc64le.
    if grep -s -e '-m elf64lppc' $f >/dev/null; then
        continue
    fi

    # Filter out candidates that don't handle PPC.
    if ! grep -s -e '-m elf64ppc' $f >/dev/null; then
        continue
    fi

    echo "Broken -m elf64ppc use in $f should handle elf64lppc."
    echo "Attempting automatic fix."

    # Attempt to fix the offended file.
    basename=`basename $f`
    dirname=`dirname $f`
    for p in $PATCH1 $PATCH2; do
        # This is an all for nothing affair.  The patch either
        # applies entirely clean, or we don't even try.
        #
        # Tentatively try either patch cleanly, and if we succeed then
        # do it for real.
        pushd $dirname 2>&1 > /dev/null
        if [ $basename = libtool.m4 ]; then
            sed s/configure/libtool.m4/ < $p | patch --dry-run --follow-symlinks -l 2>&1 >/dev/null
        else
            patch --dry-run --follow-symlinks -l < $p 2>&1 > /dev/null
        fi
        if [ $? != 0 ]; then
            echo -n "$p approach did not work for $dirname/$basename: "
            pwd
            # This approach didn't work, try the next one.
            popd 2>&1 > /dev/null
            continue
        fi

        # Seriously now...
        if [ $basename = libtool.m4 ]; then
            # Save the timestamp.
            cp -p $basename /tmp/tmp.$$
            sed s/configure/libtool.m4/ < $p | patch --follow-symlinks -l -s 2>&1 > /dev/null
            # Use the old timestamp, to avoid anyone noticing changes
            # to libtool.m4.
            touch --reference=/tmp/tmp.$$ $basename
            rm -f /tmp/tmp.$$
        else
            patch --follow-symlinks -l -s < $p 2>&1 > /dev/null
        fi
        echo "Fixed $f for ld -m ppc64le support."
        popd 2>&1 > /dev/null
        break
    done
done

rm -f $PATCH1 $PATCH2

[jsquyres]

@Zhiming-Wang Ah, ok, that explains it (i.e., how it's fixed on the power platform). The idea is that the Open MPI community tarball will contain the patch so that Power users don't have to run autogen.pl (i.e., autogen.pl is really a developer-level script -- 99% of users who install Open MPI don't run it).

@Zhiming-Wang @nysal Please look at the patch_autotools_output() subroutine in autogen.pl. This is where we apply patches -- like the one @nysal proposed in the initial description of this issue -- after we run the GNU autotools. We have several other patches in there already that you can look at as examples. I'm assuming that the libtool.m4 that Nysal's patch refers to ends up in configure itself. If that's the case, you can just add some more Perl regular expressions to the infrastructure that is there to patch configure itself (because you can't patch libtool.m4 itself -- that's both created and then used inside the execution of autoreconf -ivf). Make sense?

Pro tip: for lengthy verbatim pastes (e.g., for long shell scripts like the one you posted), you might want to use gist.github.com. It's like pastebin. You can upload the file there and then link to it here in the github issue.

[nysal]

@jsquyres Here's an intial attempt at a fix. There is a problem though, patch_autotools_output is not invoked if there is an autogen.(sh|pl) in the sub directory. So with the below patch, the configure script in ompi/contrib/vt/vt/extlib/otf and ompi/contrib/vt/vt are patched. libevent, romio and the main configure script remain unpatched. Comments ?

--- a/autogen.pl
+++ b/autogen.pl
@@ -969,6 +969,22 @@ sub patch_autotools_output {
     verbose "$indent_str"."Patching configure for IBM xlf libtool bug\n";
     $c =~ s/(\$LD -shared \$libobjs \$deplibs \$)compiler_flags( -soname \$soname)/$1linker_flags$2/g;

+    verbose "$indent_str"."Patching configure for Power Little Endian support\n";
+    my $replace_string = "x86_64-*kfreebsd*-gnu|x86_64-*linux*|powerpc*-*linux*|";
+    $c =~ s/x86_64-\*kfreebsd\*-gnu\|x86_64-\*linux\*\|ppc\*-\*linux\*\|powerpc\*-\*linux\*\|/$replace_string/g;
+    $replace_string =
+    "powerpc64le-*linux*)
+        LD=\"\${LD-ld} -m elf32lppclinux\"
+        ;;
+     powerpc64-*linux*)";
+    $c =~ s/ppc64-\*linux\*\|powerpc64-\*linux\*\)/$replace_string/g;
+    $replace_string =
+    "powerpcle-*linux*)
+        LD=\"\${LD-ld} -m elf64lppc\"
+        ;;
+     powerpc-*linux*)";
+    $c =~ s/ppc\*-\*linux\*\|powerpc\*-\*linux\*\)/$replace_string/g;
+
     open(OUT, ">configure.patched") || my_die "Can't open configure.patched";
     print OUT $c;
     close(OUT);

[Zhiming-Wang]

Yes, The fix should be add into patch_autotools_output() subroutine. The file "/usr/lib/rpm/redhat/libtool-handle-ppc64le.sh" is not guaranteed to exist all the Linux distributions which Power LE is supported.
Next, I will check the actions on Ubuntu 14.04.

[nysal]

okay, so here's the latest patch. Please review:

--- a/autogen.pl
+++ b/autogen.pl
@@ -179,11 +179,13 @@ sub process_subdir {
         print "--- Found configure.in|ac; running autoreconf...\n";
         safe_system("autoreconf -ivf");
         print "--- Patching autotools output... :-(\n";
-        patch_autotools_output($start);
     } else {
         my_die "Found subdir, but no autogen.sh or configure.in|ac to do anything";
     }

+    # Fix known issues in Autotools output
+    patch_autotools_output($start);
+
     # Ensure that we got a good configure executable.
     my_die "Did not generate a \"configure\" executable in $dir.\n"
         if (! -x "configure");
@@ -969,6 +971,22 @@ sub patch_autotools_output {
     verbose "$indent_str"."Patching configure for IBM xlf libtool bug\n";
     $c =~ s/(\$LD -shared \$libobjs \$deplibs \$)compiler_flags( -soname \$soname)/$1linker_flags$2/g;

+    verbose "$indent_str"."Patching configure for Power Little Endian support\n";
+    my $replace_string = "x86_64-*kfreebsd*-gnu|x86_64-*linux*|powerpc*-*linux*|";
+    $c =~ s/x86_64-\*kfreebsd\*-gnu\|x86_64-\*linux\*\|ppc\*-\*linux\*\|powerpc\*-\*linux\*\|/$replace_string/g;
+    $replace_string =
+    "powerpc64le-*linux*)
+        LD=\"\${LD-ld} -m elf32lppclinux\"
+        ;;
+     powerpc64-*linux*)";
+    $c =~ s/ppc64-\*linux\*\|powerpc64-\*linux\*\)/$replace_string/g;
+    $replace_string =
+    "powerpcle-*linux*)
+        LD=\"\${LD-ld} -m elf64lppc\"
+        ;;
+     powerpc-*linux*)";
+    $c =~ s/ppc\*-\*linux\*\|powerpc\*-\*linux\*\)/$replace_string/g;
+
     open(OUT, ">configure.patched") || my_die "Can't open configure.patched";
     print OUT $c;
     close(OUT);
@@ -1276,6 +1294,8 @@ foreach my $project (@{$projects}) {
 }
 safe_system($cmd);

+patch_autotools_output(".");
+
 #---------------------------------------------------------------------------

 verbose "

[jsquyres]

@nysal I think you beat me to it -- I think you fixed it in open-mpi/ompi-release#668...?

[nysal]

@jsquyres Please do review, my perl skills aren't the best. Limited testing (create tarball on a x86 system with libtool 2.4.2. Extract tarball, build and test ompi examples on powerpcle system) seems to indicate that it works.

[jsquyres]

@nysal Pro tip: if you put "Closes #396" in the commit message for your commit on open-mpi/ompi-release#668, it'll auto-close this ticket when we merge it.

[jsquyres]

open-mpi/ompi-release#668 was merged; closing this issue.