compiling error #7618

jchensw · 2021-11-02T18:13:19Z

make install
Making install in src/ucm
make[1]: Entering directory '/home/jinchen/TEST/ucx-1.11.2/src/ucm'
Making install in .
make[2]: Entering directory '/home/jinchen/TEST/ucx-1.11.2/src/ucm'
CC event/libucm_la-event.lo
CC malloc/libucm_la-malloc_hook.lo
CC mmap/libucm_la-install.lo
CC util/libucm_la-replace.lo
CC util/libucm_la-log.lo
CC util/libucm_la-reloc.lo
CC util/libucm_la-sys.lo
CC bistro/libucm_la-bistro.lo
CC bistro/libucm_la-bistro_x86_64.lo
CC bistro/libucm_la-bistro_aarch64.lo
CC bistro/libucm_la-bistro_ppc64.lo
"bistro/bistro_ppc64.c", line 157: error #1144: arithmetic on pointer to void or function type
func += 8;
^

"bistro/bistro_ppc64.c", line 158: error #1144: arithmetic on pointer to void or function type
hook += 8;
^

2 errors detected in the compilation of "bistro/bistro_ppc64.c".

jchensw · 2021-11-02T19:37:12Z

would you please take a look and give me a hint to fix it?''

Thanks,

-- Jin

dmitrygx · 2021-11-02T19:54:00Z

@jchensw this is a compilation problem in UCM/PPC64-related code.
could you verify if #7619 PR fixes the issue, please?

jchensw · 2021-11-02T21:02:25Z

@dmitrygx Thanks. it's fixed. But now I'm getting the following errors. Do you have any idea?

make[2]: Entering directory '/home/jinchen/TEST/ucx-1.11.2/src/tools/info'
CC ucx_info-build_info.o
CC ucx_info-proto_info.o
CC ucx_info-sys_info.o
CC ucx_info-tl_info.o
CC ucx_info-type_info.o
CC ucx_info-ucx_info.o
CCLD ucx_info
ucx_info-proto_info.o: In function print_ucp_info': /home/jinchen/TEST/ucx-1.11.2/src/tools/info/proto_info.c:66: undefined reference to __builtin_ppc_get_timebase'
/home/jinchen/TEST/ucx-1.11.2/src/tools/info/proto_info.c:66: undefined reference to __builtin_ppc_get_timebase' /home/jinchen/TEST/ucx-1.11.2/src/tools/info/proto_info.c:83: undefined reference to __builtin_ppc_get_timebase'
/home/jinchen/TEST/ucx-1.11.2/src/tools/info/proto_info.c:83: undefined reference to __builtin_ppc_get_timebase' ucx_info-sys_info.o: In function print_sys_info':
/home/jinchen/TEST/ucx-1.11.2/src/tools/info/sys_info.c:72: undefined reference to __builtin_ppc_get_timebase' ucx_info-sys_info.o:/home/jinchen/TEST/ucx-1.11.2/src/tools/info/sys_info.c:75: more undefined references to __builtin_ppc_get_timebase' follow
make[2]: *** [Makefile:563: ucx_info] Error 2
make[2]: Leaving directory '/home/jinchen/TEST/ucx-1.11.2/src/tools/info'
make[1]: *** [Makefile:783: install] Error 2
make[1]: Leaving directory '/home/jinchen/TEST/ucx-1.11.2/src/tools/info'
make: *** [Makefile:735: install-recursive] Error 1

dmitrygx · 2021-11-02T21:29:09Z

@dmitrygx Thanks. it's fixed. But now I'm getting the following errors. Do you have any idea?

thanks!
it seems that you're compiling UCX without glibc, right? do you use another C library - e.g. musl?
could you pool the latest changes from the PR and check if they help you?

jchensw · 2021-11-02T21:49:07Z

I'm not using gnu c library. If I do want to use it, how to specify it on the configuration step? Thanks.

dmitrygx · 2021-11-02T22:15:39Z

@jchensw e.g. specify CC=gcc when launching configure.

btw, did you check the PR without glibc?

jchensw · 2021-11-02T22:18:08Z

Not yet. I will do it right now.

jchensw · 2021-11-03T13:48:55Z

@dmitrygx It could not fix the issue. Maybe I missed something here. Would you please give me more specific suggestions? Thanks.

dmitrygx · 2021-11-03T13:51:38Z

Maybe I missed something here. Would you please give me more specific suggestions?

did you run ./autogen.sh && ./configure prior to compiling the UCX when the changes from the PR were taken?

jchensw · 2021-11-03T14:00:45Z

I checked out the master branch, and did

993 18:24 ./autogen.sh
994 18:24 ./contrib/configure-release --prefix=pwd/build/test
1013 9:40 make -j8 install

dmitrygx · 2021-11-03T14:03:39Z

do you mean that you checked out changes from #7619 instead of the master branch?

dmitrygx · 2021-11-03T14:05:29Z

$ git remote add dmitrygx https://github.com/dmitrygx/ucx.git
$ git fetch dmitrygx
$ git checkout dmitrygx/topic/ucm/bistro_ppc_fix
$ ./autogen.sh
$ ./contrib/configure-release --prefix=pwd/build/test
$ make -j8 install

jchensw · 2021-11-03T14:26:53Z

I did followed #7619, but the error is still there. So I followed your instruction one more time to be sure. And I'm still getting them

make install-am
make[2]: Entering directory '/TEST/ucx/src/tools/info'
CC ucx_info-build_info.o
CC ucx_info-proto_info.o
CC ucx_info-sys_info.o
CC ucx_info-tl_info.o
CC ucx_info-type_info.o
CC ucx_info-ucx_info.o
CCLD ucx_info
ucx_info-proto_info.o: In function print_ucp_info': /TEST/ucx/src/tools/info/proto_info.c:66: undefined reference to __builtin_ppc_get_timebase'
/TEST/ucx/src/tools/info/proto_info.c:66: undefined reference to __builtin_ppc_get_timebase' /TEST/ucx/src/tools/info/proto_info.c:83: undefined reference to __builtin_ppc_get_timebase'
/TEST/ucx/src/tools/info/proto_info.c:83: undefined reference to __builtin_ppc_get_timebase' ucx_info-sys_info.o: In function print_sys_info':
/TEST/ucx/src/tools/info/sys_info.c:73: undefined reference to __builtin_ppc_get_timebase' ucx_info-sys_info.o:/TEST/ucx/src/tools/info/sys_info.c:76: more undefined references to __builtin_ppc_get_timebase' follow

jchensw · 2021-11-03T14:27:24Z

~/TEST/ucx$git branch

(HEAD detached at dmitrygx/topic/ucm/bistro_ppc_fix)
master

jchensw · 2021-11-03T14:51:04Z

We have /usr/lib64/libc.so.6

But I'm not using gcc compiler. Then how do I let ucx know and use libc? Is there a way to specify at configuration?

Thanks,

jchensw · 2021-11-03T14:57:53Z

Here is what I have

nm /usr/lib64/libc.a | grep -i ppc_get_timebase
0000000000000000 W __ppc_get_timebase_freq

yosefe · 2021-11-03T17:29:58Z

@jchensw can you pls re-run ./autogen.sh after applying the changes in #7619 ?

jchensw · 2021-11-03T17:36:15Z

I did and here is what I got:

~/TEST/ucx$./autogen.sh

autoreconf: Entering directory `.'

autoreconf: configure.ac: not using Gettext

autoreconf: running: aclocal -I config/m4

autoreconf: configure.ac: tracing

autoreconf: running: libtoolize --copy

autoreconf: running: /usr/bin/autoconf

autoreconf: running: /usr/bin/autoheader

autoreconf: running: automake --add-missing --copy --no-force

autoreconf: Leaving directory `.'

dmitrygx · 2021-11-03T17:40:01Z

@jchensw could you pls share the output of the following command launched from UCX root directory:

$ grep "HAVE_DECL___PPC" ./config.h

wlepera · 2021-11-03T17:43:07Z

I just completed two successful builds on Power9 ppc64le, first using dmitrygx/topic/ucm/bistro_ppc_fix and again using master. Not seeing these problems in either build

jchensw · 2021-11-03T17:44:59Z

ls -ltr config.h
-rw-r--r--. 1 jinchen pppl 32449 Nov 3 10:21 config.h
~/TEST/ucx$grep "HAVE_DECL___PCC" ./config.h

No output.

dmitrygx · 2021-11-03T17:49:58Z

No output.

hmm, interesting.
it should be:

$grep "HAVE_DECL___PPC" ./config.h
#define HAVE_DECL___PPC_GET_TIMEBASE 0
#define HAVE_DECL___PPC_GET_TIMEBASE_FREQ 0

dmitrygx · 2021-11-03T17:51:15Z

@jchensw sorry, I made a mistake, it should be "HAVE_DECL___PPC" instead of "HAVE_DECL___PCC":

$grep "HAVE_DECL___PPC" ./config.h

jchensw · 2021-11-03T17:53:13Z

~/TEST/ucx$grep "HAVE_DECL___PPC" ./config.h
#define HAVE_DECL___PPC_GET_TIMEBASE 1
#define HAVE_DECL___PPC_GET_TIMEBASE_FREQ 1

yosefe · 2021-11-03T17:58:27Z

@jchensw can you pls the output of lscpu command on your machine?
@dmitrygx I think the issue is we check for decl but not the function (use AC_CHECK_FUNCS)

jchensw · 2021-11-03T18:01:45Z

~/TEST/ucx$which lscpu
/usr/bin/lscpu
~/TEST/ucx$ lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 4
Core(s) per socket: 16
Socket(s): 2
NUMA node(s): 6
Model: 2.3 (pvr 004e 1203)
Model name: POWER9, altivec supported
CPU max MHz: 3800.0000
CPU min MHz: 2300.0000
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 10240K
NUMA node0 CPU(s): 0-63
NUMA node8 CPU(s): 64-127
NUMA node252 CPU(s):
NUMA node253 CPU(s):
NUMA node254 CPU(s):
NUMA node255 CPU(s):

dmitrygx · 2021-11-03T18:02:29Z

@dmitrygx I think the issue is we check for decl but not the function (use AC_CHECK_FUNCS)

yes, it sounds like a good suggestion

dmitrygx · 2021-11-03T18:15:36Z

@jchensw could you pls pull the latest changes from the #7619 PR and check the compilation?
also, could share the output of the following command, pls:

$grep "HAVE___PPC" ./config.h

jchensw · 2021-11-03T18:21:27Z

~/TEST/ucx$ git fetch dmitrygx
From https://github.com/dmitrygx/ucx
ac6bc97..45d5367 topic/ucm/bistro_ppc_fix -> dmitrygx/topic/ucm/bistro_ppc_fix

~/TEST/ucx$ git checkout dmitrygx/topic/ucm/bistro_ppc_fix
M src/ucm/bistro/bistro_ppc64.c
Previous HEAD position was ac6bc97 CONFIG/UCS/SYS: Check _ppc_get_timebase declaration
HEAD is now at 45d5367 CONFIG/UCS/ARCH: Check for __ppc_get_timebase/__ppc_get_timbase_freq functions too

~/TEST/ucx$ git branch

(HEAD detached at dmitrygx/topic/ucm/bistro_ppc_fix)
master

~/TEST/ucx$ grep "HAVE___PPC" ./config.h

No output.

dmitrygx · 2021-11-03T18:22:16Z

~/TEST/ucx$ grep "HAVE___PPC" ./config.h

No output.

this should be checked after ./autogen && ./configure

jchensw · 2021-11-03T18:30:23Z

Here is what I have after ./autogen && ./configure

~/TEST/ucx$grep "HAVE___PPC" ./config.h
/* #undef HAVE___PPC_GET_TIMEBASE */
#define HAVE___PPC_GET_TIMEBASE_FREQ 1

jchensw · 2021-11-03T18:32:06Z

I'm still getting the same error, but this time from a different file:

make[2]: Entering directory '/home/jinchen/TEST/ucx/src/tools/perf/lib'
CC libucxperf_la-libperf.lo
CC libucxperf_la-libperf_memory.lo
CC libucxperf_la-libperf_thread.lo
CXX libucxperf_la-uct_tests.lo
CXX libucxperf_la-ucp_tests.lo
"/usr/include/sys/platform/ppc.h", line 31: error #20: identifier "__builtin_ppc_get_timebase" is undefined
return __builtin_ppc_get_timebase ();
^

"/usr/include/sys/platform/ppc.h", line 31: error #20: identifier "__builtin_ppc_get_timebase" is undefined
return __builtin_ppc_get_timebase ();
^

1 error detected in the compilation of "ucp_tests.cc".

wlepera · 2021-11-03T18:47:30Z

my env:
glibc-2.28-151.el8.ppc64le
libgcc-8.4.1-1.el8.ppc64le
gcc-8.4.1-1.el8.ppc64le
gcc-c++-8.4.1-1.el8.ppc64le

lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 176
On-line CPU(s) list: 0-175
Thread(s) per core: 4
Core(s) per socket: 22
Socket(s): 2
NUMA node(s): 8
Model: 2.3 (pvr 004e 1203)
Model name: POWER9, altivec supported
CPU max MHz: 3800.0000
CPU min MHz: 2300.0000
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 10240K
NUMA node0 CPU(s): 0-87
NUMA node8 CPU(s): 88-175
NUMA node250 CPU(s):
NUMA node251 CPU(s):
NUMA node252 CPU(s):
NUMA node253 CPU(s):
NUMA node254 CPU(s):
NUMA node255 CPU(s):

jchensw · 2021-11-03T18:53:21Z

I'm using nvhpc compiler, not gcc compiler, but we do have glibc.

dmitrygx · 2021-11-03T19:07:05Z

I'm using nvhpc compiler, not gcc compiler, but we do have glibc.

thanks for the additional information.

@jchensw could you pull the latest changes from my PR and try "./autogen && ./configure && make" again?
thanks in advance!

wlepera · 2021-11-03T19:47:03Z

@jchensw which nvhpc version are you using?

jchensw · 2021-11-03T21:40:41Z

Sorry for the late response.

The errors are still there :

1028 17:25 git fetch dmitrygx
remote: Enumerating objects: 288, done.
remote: Counting objects: 100% (146/146), done.
remote: Compressing objects: 100% (35/35), done.
remote: Total 83 (delta 55), reused 63 (delta 41), pack-reused 0
Unpacking objects: 100% (83/83), 11.64 KiB | 170.00 KiB/s, done.
From https://github.com/dmitrygx/ucx
45d5367..96ac8b5 topic/ucm/bistro_ppc_fix -> dmitrygx/topic/ucm/bistro_ppc_fix

[new branch] topic/ucp/discard_invalid_ep -> dmitrygx/topic/ucp/discard_invalid_ep

1029 17:26 git checkout dmitrygx/topic/ucm/bistro_ppc_fix
M src/ucm/bistro/bistro_ppc64.c
Previous HEAD position was 45d5367 CONFIG/UCS/ARCH: Check for __ppc_get_timebase/__ppc_get_timbase_freq functions too
HEAD is now at 96ac8b5 PERF/UCP: Include UCS headers under extern 'C'

1030 17:26 git branch

(HEAD detached at dmitrygx/topic/ucm/bistro_ppc_fix)
master

1031 17:26 ls -ltr src/ucm/bistro/bistro_ppc64.*
1032 17:27 diff src/ucm/bistro/bistro_ppc64.c src/ucm/bistro/bistro_ppc64.c_new
1033 17:27 ./autogen.sh
1034 17:28 history | grep trib
1035 17:28 ./contrib/configure-release --prefix=pwd/build/test
Making install in lib
make[2]: Entering directory '/home/jinchen/TEST/ucx/src/tools/perf/lib'
CC libucxperf_la-libperf.lo
CC libucxperf_la-libperf_memory.lo
CC libucxperf_la-libperf_thread.lo
CXX libucxperf_la-uct_tests.lo
CXX libucxperf_la-ucp_tests.lo
"/usr/include/sys/platform/ppc.h", line 31: error Add performance testing framework #20: identifier "__builtin_ppc_get_timebase" is undefined
return __builtin_ppc_get_timebase ();
^
"/usr/include/sys/platform/ppc.h", line 31: error Add performance testing framework #20: identifier "__builtin_ppc_get_timebase" is undefined
return __builtin_ppc_get_timebase ();
^
1 error detected in the compilation of "ucp_tests.cc".

jchensw · 2021-11-03T21:51:00Z

@jchensw which nvhpc version are you using?

I'm using

nvhpc/21.5 2) cudatoolkit/11.4 3) cmake/3.19.7

dmitrygx · 2021-11-03T22:30:24Z

@jchensw could you try the latest state of the PR, pls?

jchensw · 2021-11-03T22:54:28Z

I did a fresh start:
1066 18:40 git clone https://github.com/openucx/ucx.git
1067 18:40 cd ucx
1071 18:40 git remote add dmitrygx https://github.com/dmitrygx/ucx.git
1072 18:40 git fetch dmitrygx
1074 18:40 git checkout dmitrygx/topic/ucm/bistro_ppc_fix
1076 18:41 ./autogen.sh
1078 18:49 ./contrib/configure-release --prefix=pwd/build/test
1080 18:51 make -j8 install

and I'm still getting the same error:

Making install in lib
make[2]: Entering directory '/home/jinchen/TEST/ucx/src/tools/perf/lib'
CC libucxperf_la-libperf.lo
CC libucxperf_la-libperf_memory.lo
CC libucxperf_la-libperf_thread.lo
CXX libucxperf_la-uct_tests.lo
CXX libucxperf_la-ucp_tests.lo
"/usr/include/sys/platform/ppc.h", line 31: error #20: identifier "__builtin_ppc_get_timebase" is undefined
return __builtin_ppc_get_timebase ();
^
"/usr/include/sys/platform/ppc.h", line 31: error #20: identifier "__builtin_ppc_get_timebase" is undefined
return __builtin_ppc_get_timebase ();
^
1 error detected in the compilation of "ucp_tests.cc".

jchensw · 2021-11-03T22:56:26Z

Should I try #7620?

dmitrygx · 2021-11-04T15:18:08Z

@jchensw we got an answer from nvc++ compiler folks that it doesn't fully support some built-ins, especially on PPC.
can you pls use gcc/clang instead?

wlepera · 2021-11-04T15:22:07Z

fyi, this is my build config (from the tail of the configure output):

configure: =========================================================
configure: UCX build configuration:
configure:         Build prefix:   /smpi_dev/lepera/ucx_7618a/install
configure:    Configuration dir:   ${prefix}/etc/ucx
configure:   Preprocessor flags:   -DCPU_FLAGS="" -I${abs_top_srcdir}/src -I${abs_top_builddir} -I${abs_top_builddir}/src
configure:           C compiler:   gcc -O3 -g -Wall -Werror -funwind-tables -Wno-missing-field-initializers -Wno-unused-parameter -Wno-unused-label -Wno-long-long -Wno-endif-labels -Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Winvalid-pch -Wno-pointer-sign -Werror-implicit-function-declaration -Wno-format-zero-length -Wnested-externs -Wshadow -Werror=declaration-after-statement
configure:         C++ compiler:   g++ -O3 -g -Wall -Werror -funwind-tables -Wno-missing-field-initializers -Wno-unused-parameter -Wno-unused-label -Wno-long-long -Wno-endif-labels -Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Winvalid-pch
configure:         Multi-thread:   disabled
configure:         NUMA support:   enabled
configure:            MPI tests:   disabled
configure:          VFS support:   no
configure:        Devel headers:   no
configure: io_demo CUDA support:   no
configure:             Bindings:   < >
configure:          UCS modules:   < >
configure:          UCT modules:   < ib rdmacm cma knem >
configure:         CUDA modules:   < >
configure:         ROCM modules:   < >
configure:           IB modules:   < >
configure:          UCM modules:   < >
configure:         Perf modules:   < >
configure: =========================================================

jchensw · 2021-11-04T15:47:05Z

Thanks. gcc works.

dmitrygx · 2021-11-04T15:53:07Z

Thanks. gcc works.

thanks for confirming it.

dmitrygx mentioned this issue Nov 2, 2021

UCM/UCS/CONFIG: Fix compile errors for ppc64 #7619

Merged

dmitrygx closed this as completed in #7619 Nov 4, 2021

compiling error #7618

compiling error #7618

Comments

jchensw commented Nov 2, 2021

jchensw commented Nov 2, 2021

dmitrygx commented Nov 2, 2021

jchensw commented Nov 2, 2021

dmitrygx commented Nov 2, 2021

jchensw commented Nov 2, 2021

dmitrygx commented Nov 2, 2021

jchensw commented Nov 2, 2021

jchensw commented Nov 3, 2021

dmitrygx commented Nov 3, 2021 • edited Loading

jchensw commented Nov 3, 2021

dmitrygx commented Nov 3, 2021

dmitrygx commented Nov 3, 2021

jchensw commented Nov 3, 2021 • edited Loading

jchensw commented Nov 3, 2021

jchensw commented Nov 3, 2021

jchensw commented Nov 3, 2021

yosefe commented Nov 3, 2021

jchensw commented Nov 3, 2021 • edited Loading

dmitrygx commented Nov 3, 2021 • edited Loading

wlepera commented Nov 3, 2021

jchensw commented Nov 3, 2021 • edited Loading

dmitrygx commented Nov 3, 2021

dmitrygx commented Nov 3, 2021

jchensw commented Nov 3, 2021

yosefe commented Nov 3, 2021

jchensw commented Nov 3, 2021

dmitrygx commented Nov 3, 2021

dmitrygx commented Nov 3, 2021

jchensw commented Nov 3, 2021

dmitrygx commented Nov 3, 2021

jchensw commented Nov 3, 2021

jchensw commented Nov 3, 2021

wlepera commented Nov 3, 2021

jchensw commented Nov 3, 2021

dmitrygx commented Nov 3, 2021

wlepera commented Nov 3, 2021

jchensw commented Nov 3, 2021

jchensw commented Nov 3, 2021

dmitrygx commented Nov 3, 2021

jchensw commented Nov 3, 2021

jchensw commented Nov 3, 2021

dmitrygx commented Nov 4, 2021

wlepera commented Nov 4, 2021

jchensw commented Nov 4, 2021

dmitrygx commented Nov 4, 2021

dmitrygx commented Nov 3, 2021 •

edited

Loading

jchensw commented Nov 3, 2021 •

edited

Loading

jchensw commented Nov 3, 2021 •

edited

Loading

dmitrygx commented Nov 3, 2021 •

edited

Loading

jchensw commented Nov 3, 2021 •

edited

Loading