[mmlspark] JVM crash caused by LGBM_BoosterSaveModelToStringSWIG

[REASY]

Environment info

Operating System: Windows 10 x64, version 10.0.18362.388
Java: Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

CPU/GPU model: AMD Ryzen 7 2700X

LightGBM version or commit hash:

• "com.microsoft.ml.spark" %% "mmlspark" % "0.18.1"
• "com.microsoft.ml.lightgbm" % "lightgbmlib" % "2.2.350" via https://github.com/Azure/mmlspark/blob/v0.18.1/build.sbt#L27

Error message

[info] [LightGBM] [Warning] Starting from the 2.1.2 version, default value for the "boost_from_average" parameter in "binary" objective is true.
[info] This may cause significantly different results comparing to the previous versions of LightGBM.
[info] Try to set boost_from_average=false, if your old models produce bad results
[info] [LightGBM] [Info] Number of positive: 232, number of negative: 8245
[info] [LightGBM] [Info] Total Bins 5466
[info] [LightGBM] [Info] Number of data: 8477, number of used features: 41
[info] [LightGBM] [Info] [binary:BoostFromScore]: pavg=0.027368 -> initscore=-3.570625
[info] [LightGBM] [Info] Start training from score -3.570625
[info] #
[info] # A fatal error has been detected by the Java Runtime Environment:
[info] #
[info] #  EXCEPTION_UNCAUGHT_CXX_EXCEPTION (0xe06d7363) at pc=0x00007ff8eee3a839, pid=18188, tid=0x0000000000003da4
[info] #
[info] # JRE version: Java(TM) SE Runtime Environment (8.0_221-b11) (build 1.8.0_221-b11)
[info] # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.221-b11 mixed mode windows-amd64 compressed oops)
[info] # Problematic frame:
[info] # C  [KERNELBASE.dll+0x3a839]
[info] #
[info] # Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
[info] #
[info] # An error report file with more information is saved as:
[info] # C:\repos\SparkPlayground\hs_err_pid18188.log
[info] #
[info] # If you would like to submit a bug report, please visit:
[info] #   http://bugreport.java.com/bugreport/crash.jsp
[info] # The crash happened outside the Java Virtual Machine in native code.
[info] # See problematic frame for where to report the bug.
[info] #
[error] Nonzero exit code returned from runner: 1
[error] (Compile / runMain) Nonzero exit code returned from runner: 1
[error] Total time: 27 s, completed Oct 9, 2019 4:09:47 AM

Link to hs_err_pid18188.log: https://pastebin.com/8TgKB61c

Reproducible examples

Steps to reproduce

1. Clone https://github.com/REASY/spark-playground/
2. Switch to branch lightgbm-0.18.1-crash-repro
3. Save to the local disk input file part-00000-711560fe-8fdd-4777-a379-b52996fd212d-c000.gz.parquet: https://drive.google.com/open?id=1iE2ptX9O4x5vn1KjDdZYVuSIuLgzyLbJ
4. Replace the path in https://github.com/REASY/spark-playground/blob/lightgbm-0.18.1-crash-repro/src/main/scala/spark/playground/LigthGbmUsage.scala#L13 by your local path (sorry for this, it could be done as app args)
5. Run sbt "runMain spark.playground.LigthGbmUsage" in console (expecting that you have installed SBT
6. From time to time it crashes

[StrikerRUS]

@imatiach-msft Please take a look

[imatiach-msft]

@REASY @StrikerRUS sorry about the trouble you are having. Indeed, this was a bug and it should already be fixed with this PR:
#2364
This issue should no longer appear in the latest master branch, although it is still an issue in 0.18.1 since it doesn't include the fix yet. Sorry about the inconvenience.

[imatiach-msft]

also see the corresponding mmlspark fix:
microsoft/SynapseML#676

[REASY]

Thanks, @imatiach-msft. I'll try with snapshot version!

[REASY]

Confirming that using latest master version 0.18.1-30-a44dafd4-SNAPSHOT has solved it. Thanks again!

[StrikerRUS]

@REASY Glad to hear that your problem has been solved! Thank you for the feedback!

@imatiach-msft Thanks a lot!