Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The higher version dependencies when build lightGBM #1858

Closed
xiaoxiaofengzi opened this issue Nov 20, 2018 · 5 comments
Closed

The higher version dependencies when build lightGBM #1858

xiaoxiaofengzi opened this issue Nov 20, 2018 · 5 comments

Comments

@xiaoxiaofengzi
Copy link

/lib64/libstdc++.so.6: version GLIBCXX_3.4.20 not found
When I build LightGBM project, the lib_lightgbm.so needs the higher version dependencies. What can I do when I don't have the root authority of Linux machine? Thank you very much!

@StrikerRUS
Copy link
Collaborator

Hi @xiaoxiaofengzi !

Can you please provide more information about your environment from GitHub issue template?

Also, it's not clear what do you mean by

When I build LightGBM project ...

Are you trying to build LightGBM library from sources or your research project with LightGBM?

@xiaoxiaofengzi
Copy link
Author

Hi @StrikerRUS !
What I do is that I train a LightGBM classification model by using the scala version of mmlspark. And the version of mmlspark is com.microsoft.ml.spark:mmlspark_2.11:0.14.dev9+1.g5783ce91. However, when I solved the problem about the the version of GLIBC, a new problem about the version of GLIBCXX appeared.
The wrong details are as follows:

18/11/19 15:11:26 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 3.0 (TID 30, hadoop-188-10, executor 1): java.lang.UnsatisfiedLinkError: /data/deploy/opt/data/hadoop/tmp/hadoop-deploy/nm-local-dir/usercache/deploy/appcache/application_1542027375692_2593/container_e23_1542027375692_2593_01_000002/tmp/mml-natives6225024011850668628/lib_lightgbm.so: /lib64/libstdc++.so.6: version 'GLIBCXX_3.4.20' not found (required by /data/deploy/opt/data/hadoop/tmp/hadoop-deploy/nm-local-dir/usercache/deploy/appcache/application_1542027375692_2593/container_e23_1542027375692_2593_01_000002/tmp/mml-natives6225024011850668628/lib_lightgbm.so) at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) at java.lang.Runtime.load0(Runtime.java:809) at java.lang.System.load(System.java:1086) at com.microsoft.ml.spark.NativeLoader.loadLibraryByName(NativeLoader.java:59) at com.microsoft.ml.spark.LightGBMClassifier$$anonfun$1.apply(LightGBMClassifier.scala:60) at com.microsoft.ml.spark.LightGBMClassifier$$anonfun$1.apply(LightGBMClassifier.scala:60) at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$6.apply(objects.scala:196) at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$6.apply(objects.scala:193) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335) at java.lang.Thread.run(Thread.java:748)

I don't know what can I do to solve the new problem. Could you understand me? Thank you very much for your attention.

@StrikerRUS
Copy link
Collaborator

Thank you @xiaoxiaofengzi for the clarification!
Unfortunately this is a known problem caused by that MMLSpark uses old and broken (in terms of dependencies) version of LightGBM.

You can track this PR and these issues 1, 2 to know when the problem will be solved on the Spark-package side. I think that you can copy-paste your log there to hurry them gently.
From the our side starting from 2.2.2 version we automatically guarantee GLIBC <= 2.14 and GLIBCXX <= 3.4.19:

https://github.com/Microsoft/LightGBM/blob/a694712b7fb86cd532eea2c1781b58d4ba58436a/helpers/check_dynamic_dependencies.py#L18-L23
https://github.com/Microsoft/LightGBM/blob/a694712b7fb86cd532eea2c1781b58d4ba58436a/helpers/check_dynamic_dependencies.py#L25-L31

Thanks for your patience and sorry for the inconvenience!

@StrikerRUS
Copy link
Collaborator

This issue should be solved as master MMLSpark is using LightGBM version 2.2.2 now.

@wangwei420625
Copy link

Lost task 0.0 in stage 8.0 (TID 55, xydw10, executor 1): java.lang.UnsatisfiedLinkError: /data7/yarn/nm/usercache/hdfs/appcache/application_1558442740400_53211/container_e67_1558442740400_53211_01_000002/tmp/mml-natives3056555705600509374/lib_lightgbm.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /data7/yarn/nm/usercache/hdfs/appcache/application_1558442740400_53211/container_e67_1558442740400_53211_01_000002/tmp/mml-natives3056555705600509374/lib_lightgbm.so)

@lock lock bot locked as resolved and limited conversation to collaborators Mar 11, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants