Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OpenJDK Regression suite "jdk_util" throws Segmentation Error before running tests on MacOS #5399

Closed
adamfarley opened this issue Apr 8, 2019 · 20 comments · Fixed by #7104
Closed

OpenJDK Regression suite "jdk_util" throws Segmentation Error before running tests on MacOS #5399

adamfarley opened this issue Apr 8, 2019 · 20 comments · Fixed by #7104
Labels
AQA Test failures from AdoptOpenJDK Quality Assurance test set comp:vm test failure
Milestone
Release 0.17 (Jav...

Comments

@adamfarley
Copy link
Contributor

adamfarley commented Apr 8, 2019 •
edited by pshipton
Loading

Failure link

https://ci.adoptopenjdk.net/job/openjdk11_j9_openjdktest_x86-64_macos_macosXL/46/console
https://ci.adoptopenjdk.net/job/openjdk12_j9_openjdktest_x86-64_macos_macosXL/25/console
https://ci.adoptopenjdk.net/job/openjdk11_j9_openjdktest_x86-64_macos/213/console

Optional info

  • intermittent failure: yes, but seems frequent.
  • regression or new test: regression.
  • if regression, what are the last passing / first failing public SHAs (OpenJ9, OMR, JCL) :
    Last pass is
OpenJ9   - 8cbe2695f
OMR      - 9f86693a
JCL      - 5650a7164d

First failure is

OpenJ9   - 127ba34f1
OMR      - 5ceecf19
JCL      - ff6f49a4ff

8cbe269...127ba34
eclipse-openj9/openj9-omr@9f86693...5ceecf1
ibmruntimes/openj9-openjdk-jdk11@5650a71...ff6f49a

Failure output

01:14:17  ===============================================
01:14:17  Running test jdk_util_0 ...
01:14:17  ===============================================
01:14:17  jdk_util_0 Start Time: Sat Apr  6 17:14:15 2019 Epoch Time (ms): 1554596055188
01:14:17  "/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:destroyAll; "/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:groupAccess,destroyAll; echo "cache cleanup done";
01:14:17  JVMSHRC005I No shared class caches available
01:14:17  JVMSHRC005I No shared class caches available
01:14:17  cache cleanup done
01:14:17  variation: NoOptions
01:14:17  JVM_OPTIONS: -Xnocompressedrefs 
01:14:17  { itercnt=1; \
01:14:17  	mkdir -p "/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15545846908321/jdk_util_0"; \
01:14:17  	cd "/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15545846908321/jdk_util_0"; \
01:14:17  	"/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdkbinary/j2sdk-image/bin/java" -Xmx512m -jar "/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/openjdk_regression/jtreg/lib/jtreg.jar" \
01:14:17  	-agentvm -a -ea -esa -v:fail,error,time,nopass -retain:fail,error -ignore:quiet -timeoutFactor:8 -xml:verify -concurrency:2  -vmoptions:"-Xmx512m -Xnocompressedrefs " \
01:14:17  	-w "/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/openjdk_regression/work" \
01:14:17  	-r "/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/openjdk_regression/report" \
01:14:17  	-jdk:"/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdkbinary/j2sdk-image/bin/.." \
01:14:17  	-exclude:"/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdk-tests/TestConfig/scripts/testKitGen/../../../openjdk_regression/openjdk-jdk/test/jdk/ProblemList.txt" \
01:14:17  	-exclude:"/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/openjdk_regression/ProblemList_openjdk11-openj9.txt" \
01:14:17  	"/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdk-tests/TestConfig/scripts/testKitGen/../../../openjdk_regression/openjdk-jdk/test/jdk:jdk_util"; \
01:14:17  	if [ $? -eq 0 ] ; then echo ""; echo "jdk_util_0""_PASSED"; echo ""; cd /Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdk-tests/TestConfig/scripts/testKitGen/../../..;  else echo ""; echo "jdk_util_0""_FAILED"; echo ""; fi; } 2>&1 | tee -a "/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15545846908321/TestTargetResult";
01:14:32  XML output with verification to /Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/jvmtest/openjdk_regression/work
01:20:46  [2019-04-06 17:20:33,966] Agent[3]: stderr: Unhandled exception
01:20:46  [2019-04-06 17:20:33,966] Agent[3]: stderr: Type=Segmentation error vmState=0x00020012
01:20:46  [2019-04-06 17:20:33,966] Agent[3]: stderr: J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
01:20:46  [2019-04-06 17:20:33,966] Agent[3]: stderr: Handler1=0000000006692190 Handler2=00000000068AC3B0 InaccessibleAddress=0000000100000200
01:20:46  [2019-04-06 17:20:33,966] Agent[3]: stderr: RDI=00000000B0491E80 RSI=0000000000010000 RAX=0000000000000000 RBX=00000000B0491E80
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: RCX=0000000100000110 RDX=00007FE90237C668 R8=000000005CA94251 R9=0000000000000001
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: R10=00000000085E04D7 R11=00007FE8FA3C4447 R12=0000000000000001 R13=00007FE902810420
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: R14=00000000B0491E88 R15=00000000065D1C28
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: RIP=000000000841F37E GS=0000 FS=0000 RSP=00000000B0491E70
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: RFlags=0000000000010202 CS=002B RBP=00000000B0491E70 ERR=0000020000000004
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: TRAPNO=000000040000000E CPU=0200000000040000 FAULTVADDR=0000000100000200
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: XMM1 43e0000000000000 (f: 0.000000, d: 9.223372e+18)
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: XMM2 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: XMM3 bf69215900000000 (f: 0.000000, d: -3.067659e-03)
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: XMM4 bed3c699007f547f (f: 8344703.000000, d: -4.714911e-06)
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: XMM5 3fcef0af00000000 (f: 0.000000, d: 2.417201e-01)
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: XMM6 bf69215900000000 (f: 0.000000, d: -3.067659e-03)
01:20:46  [2019-04-06 17:20:33,967] Agent[3]: stderr: XMM7 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: XMM8 01000a0000001900 (f: 6400.000000, d: 7.308923e-304)
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: XMM9 a19dfa2392e2a66d (f: 2464327168.000000, d: -9.377602e-147)
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: XMM11 ca62c1d6ca62c1d6 (f: 3395469824.000000, d: -2.193092e+50)
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: XMM12 bda2d83231300000 (f: 825229312.000000, d: -8.569494e-12)
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: XMM13 3c7793616c256dc2 (f: 1814392320.000000, d: 2.044866e-17)
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: XMM14 3c04f79b73a00000 (f: 1939865600.000000, d: 1.420794e-19)
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: Module=/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdkbinary/j2sdk-image/lib/default/libj9gc29.dylib
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: Module_base_address=0000000008400000 Symbol=_ZN20GC_ClassHeapIterator9nextClassEv
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: Symbol_address=000000000841F370
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: Target=2_90_20190406_161 (Mac OS X 10.10.5)
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: CPU=amd64 (2 logical CPUs) (0x80000000 RAM)
01:20:46  [2019-04-06 17:20:33,969] Agent[3]: stderr: ----------- Stack Backtrace -----------
01:20:46  [2019-04-06 17:20:33,969] Agent[3]: stderr: ---------------------------------------
@DanHeidinga
Copy link
Member

Crash in the GC iterating ClassHeapIterator.nextClassEv

01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: Module=/Users/jenkins/workspace/openjdk11_j9_openjdktest_x86-64_macos_macosXL/openjdkbinary/j2sdk-image/lib/default/libj9gc29.dylib
01:20:46  [2019-04-06 17:20:33,968] Agent[3]: stderr: Module_base_address=0000000008400000 Symbol=_ZN20GC_ClassHeapIterator9nextClassEv

fyi @dmitripivkine

@dmitripivkine
Copy link
Contributor

There is no results stored in these jobs. Would somebody help to get system core for this failure please?

@dmitripivkine
Copy link
Contributor

It fails different places, not just class iterating:

https://ci.adoptopenjdk.net/job/openjdk12_j9_openjdktest_x86-64_macos_macosXL/25/console

19:34:24  [2019-04-06 10:34:11,089] Agent[2]: stderr: Module_base_address=000000000A64E000 Symbol=_ZN27MM_MarkingSchemeRootClearer17doStringTableSlotEPP8J9ObjectP20GC_HashTableIterator

https://ci.adoptopenjdk.net/job/openjdk11_j9_openjdktest_x86-64_macos/213/console

06:57:11  [2019-04-06 21:56:56,786] Agent[3]: stderr: Module_base_address=000000000805B000 Symbol=pool_newElement

@JasonFengJ9
Copy link
Member

Notes from chatting with @dmitripivkine
First two links crashed at libj9gc29.dylib which are related to VM structure scanning according to vmState code.
Third link crashed at libj9vm29.dylib.

@adamfarley
Copy link
Contributor Author

I presumed a connection based on their shared moment of failure. It happens during jdk_util test startup, before it has declared its running any tests.

@dmitripivkine
Copy link
Contributor

dmitripivkine commented Apr 10, 2019 •
edited
Loading

I was able to reproduce crash once however system core was not stored in results as well as was not discovered on machine directly in preserved workspace. So it is useless. It is new place of crash _ZN22VM_BytecodeInterpreter3runEP10J9VMThread:

15:49:28  [2019-04-09 15:49:16,772] Agent[6]: stderr: Unhandled exception
15:49:28  [2019-04-09 15:49:16,772] Agent[6]: stderr: Type=Segmentation error vmState=0x00000000
15:49:28  [2019-04-09 15:49:16,772] Agent[6]: stderr: J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
15:49:28  [2019-04-09 15:49:16,772] Agent[6]: stderr: Handler1=000000000D2A2050 Handler2=000000000C8F13B0 InaccessibleAddress=0000000000000004
15:49:28  [2019-04-09 15:49:16,772] Agent[6]: stderr: RDI=000000001BA85B58 RSI=00007000089248A8 RAX=000000001BA851C0 RBX=00007000089248B8
15:49:28  [2019-04-09 15:49:16,772] Agent[6]: stderr: RCX=000000001BA85B40 RDX=FFFFFFFE00000000 R8=000000001B926A00 R9=000000001B926C08
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: R10=00007000089248D0 R11=00007FA1858EAF54 R12=00007000089248C0 R13=00007000089248B0
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: R14=0000000000000004 R15=00007000089248B0
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: RIP=000000000D231921 GS=0000 FS=0000 RSP=00007000089245B0
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: RFlags=0000000000010202 CS=002B RBP=0000700008924890 ERR=0000000400000004
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: TRAPNO=000000040000000E CPU=0004000000040000 FAULTVADDR=0000000000000004
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: XMM0 0000000000000001 (f: 1.000000, d: 4.940656e-324)
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: XMM2 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: XMM3 bf40737600000000 (f: 0.000000, d: -5.020453e-04)
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: XMM4 be80eba05c84b5f6 (f: 1552201216.000000, d: -1.260669e-07)
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: XMM5 bf9c63d000000000 (f: 0.000000, d: -2.772450e-02)
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: XMM6 000fffffffffffff (f: 4294967296.000000, d: 2.225074e-308)
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: XMM7 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:49:28  [2019-04-09 15:49:16,773] Agent[6]: stderr: XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: Module=/Users/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/lib/compressedrefs/libj9vm29.dylib
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: Module_base_address=000000000D200000 Symbol=_ZN22VM_BytecodeInterpreter3runEP10J9VMThread
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: Symbol_address=000000000D2102C0
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: Target=2_90_20190408_198 (Mac OS X 10.13.6)
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: ----------- Stack Backtrace -----------
15:49:28  [2019-04-09 15:49:16,774] Agent[6]: stderr: ---------------------------------------

@dmitripivkine
Copy link
Contributor

Also I noticed another (much more common) failure:

16:43:21  [2019-04-09 16:43:21,050] Agent[4]: stderr: java(91031,0x7000051f6000) malloc: *** error for object 0x7f866d8a0e00: incorrect checksum for freed object - object was probably modified after being freed.
16:43:21  [2019-04-09 16:43:21,050] Agent[4]: stderr: *** set a breakpoint in malloc_error_break to debug
16:47:47  [2019-04-09 16:47:38,724] Agent[0]: stderr: Timeout reached: 60000
16:47:47  [2019-04-09 16:47:38,724] Agent[0]: stderr: Process is not alive!

@JasonFengJ9
Copy link
Member

JasonFengJ9 commented Apr 10, 2019 •
edited
Loading

Captured a core and got following stacktrace (similar with 3rd link in this issue):

[2019-04-10 17:19:36,460] Agent[15]: stderr: Type=Segmentation error vmState=0x00000000
[2019-04-10 17:19:36,460] Agent[15]: stderr: J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2019-04-10 17:19:36,460] Agent[15]: stderr: Handler1=000000000B68E2D0 Handler2=000000000B880EB0 InaccessibleAddress=00007FEB24047A5C
[2019-04-10 17:19:36,460] Agent[15]: stderr: RDI=00007FEB06006D00 RSI=0000000007FFFFFF RAX=00000000FFFFFFFE RBX=00007FEB04047A20
[2019-04-10 17:19:36,460] Agent[15]: stderr: RCX=0000000000000000 RDX=0000000000000004 R8=0000000000000000 R9=0000000000000002
[2019-04-10 17:19:36,460] Agent[15]: stderr: R10=0000000000000000 R11=0000700009FDA7E0 R12=00007FEB06006D90 R13=0000000000000000
[2019-04-10 17:19:36,460] Agent[15]: stderr: R14=00007FEB04047A2C R15=00007FEB06006D00
[2019-04-10 17:19:36,460] Agent[15]: stderr: RIP=000000000B7CBA72 GS=0000 FS=0000 RSP=0000700009FDA350
[2019-04-10 17:19:36,460] Agent[15]: stderr: RFlags=0000000000010A07 CS=002B RBP=0000700009FDA390 ERR=24047A5C00000006
[2019-04-10 17:19:36,460] Agent[15]: stderr: TRAPNO=000000060000000E CPU=7A5C000000060000 FAULTVADDR=00007FEB24047A5C
[2019-04-10 17:19:36,460] Agent[15]: stderr: XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-04-10 17:19:36,460] Agent[15]: stderr: XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-04-10 17:19:36,460] Agent[15]: stderr: XMM2 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2019-04-10 17:19:36,460] Agent[15]: stderr: XMM3 bf42397808000000 (f: 134217728.000000, d: -5.561672e-04)
[2019-04-10 17:19:36,460] Agent[15]: stderr: XMM4 be84c4151be6e5b4 (f: 468116928.000000, d: -1.547184e-07)
[2019-04-10 17:19:36,460] Agent[15]: stderr: XMM5 bfc9f6c300000000 (f: 0.000000, d: -2.028431e-01)
[2019-04-10 17:19:36,460] Agent[15]: stderr: XMM6 000fffffffffffff (f: 4294967296.000000, d: 2.225074e-308)
[2019-04-10 17:19:36,460] Agent[15]: stderr: XMM7 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
[2019-04-10 17:19:36,461] Agent[15]: stderr: XMM8 0b0a090803020100 (f: 50462976.000000, d: 1.733947e-255)
[2019-04-10 17:19:36,461] Agent[15]: stderr: XMM9 ffffffffffffffff (f: 4294967296.000000, d: nan)
[2019-04-10 17:19:36,461] Agent[15]: stderr: XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-04-10 17:19:36,461] Agent[15]: stderr: XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-04-10 17:19:36,461] Agent[15]: stderr: XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-04-10 17:19:36,461] Agent[15]: stderr: XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-04-10 17:19:36,461] Agent[15]: stderr: XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-04-10 17:19:36,461] Agent[15]: stderr: XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-04-10 17:19:36,461] Agent[15]: stderr: Module=/ivm/macos/macbuilds/jdk11-0409/openj9-openjdk-jdk11/build/macosx-x86_64-normal-server-release/images/jdk/lib/compressedrefs/libj9vm29.dylib

frame #5: 0x000000000b87faad libj9prt29.dylib`omrsig_protect(portLibrary=0x000000000a562348, fn=<unavailable>, fn_arg=0x0000700009fd8128, handler=<unavailable>, handler_arg=0x0000000000000000, flags=<unavailable>, result=0x0000700009fd8118) at omrsignal.c:384:12
    frame #6: 0x000000000b8ce306 libj9dmp29.dylib`runDumpAgent [inlined] runDumpFunction(agent=<unavailable>, label=<unavailable>, context=0x0000700009fd85b0) at dmpagent.c:2850:21 [opt]
    frame #7: 0x000000000b8ce2bb libj9dmp29.dylib`runDumpAgent(vm=0x00007feb0580b820, agent=<unavailable>, context=0x0000700009fd85b0, state=<unavailable>, detail="", timeNow=<unavailable>) at dmpagent.c:2776 [opt]
    frame #8: 0x000000000b8e57b1 libj9dmp29.dylib`triggerDumpAgents(vm=0x00007feb0580b820, self=0x0000000016f0cc00, eventFlags=<unavailable>, eventData=0x0000000000000000) at trigger.c:1028:7 [opt]
    frame #9: 0x000000000b68f3a9 libj9vm29.dylib`generateDiagnosticFiles(portLibrary=<unavailable>, userData=<unavailable>) at gphandle.c:1170:3 [opt]
    frame #10: 0x000000000b87faad libj9prt29.dylib`omrsig_protect(portLibrary=0x000000000a562348, fn=<unavailable>, fn_arg=0x0000700009fd8e70, handler=<unavailable>, handler_arg=0x0000700009fd8ed0, flags=<unavailable>, result=0x0000700009fd8e68) at omrsignal.c:384:12
    frame #11: 0x000000000b68e603 libj9vm29.dylib`vmSignalHandler(portLibrary=0x000000000a562348, gpType=<unavailable>, gpInfo=<unavailable>, userData=<unavailable>) at gphandle.c:841:2 [opt]
    frame #12: 0x000000000b68e313 libj9vm29.dylib`structuredSignalHandler(portLibrary=<unavailable>, gpType=<unavailable>, gpInfo=<unavailable>, userData=<unavailable>) at gphandle.c:634:9 [opt]
    frame #13: 0x000000000b880ff6 libj9prt29.dylib`masterSynchSignalHandler(signal=11, sigInfo=<unavailable>, contextInfo=0x0000700009fda298) at omrsignal.c:1001:14
    frame #14: 0x00007fff6d6b9b5d libsystem_platform.dylib`_sigtramp + 29
    frame #15: 0x000000000b7cba73 libj9vm29.dylib`pool_newElement(pool=0x00007feb06006d00) at pool.c:521:2
    frame #16: 0x000000000b6be1bb libj9vm29.dylib`allocateVirtualMemorySegmentInListInternal [inlined] allocateMemorySegmentListEntry(segmentList=<unavailable>) at segment.c:160:37 [opt]
    frame #17: 0x000000000b6be1b3 libj9vm29.dylib`allocateVirtualMemorySegmentInListInternal(javaVM=0x00007feb0580b820, segmentList=0x00007feb06006c50, size=1024, type=65536, vmemParams=0x0000000000000000, memoryCategory=2) at segment.c:374 [opt]
    frame #18: 0x000000000b6bdabb libj9vm29.dylib`allocateMemorySegmentInList [inlined] allocateFixedMemorySegmentInList(javaVM=0x00007feb0580b820, segmentList=<unavailable>, size=<unavailable>, type=65536, memoryCategory=2) at segment.c:346:9 [opt]
    frame #19: 0x000000000b6bda2a libj9vm29.dylib`allocateMemorySegmentInList(javaVM=0x00007feb0580b820, segmentList=0x0000000009fda4b8, size=616, type=65536, memoryCategory=2) at segment.c:298 [opt]
    frame #20: 0x000000000b67dc6c libj9vm29.dylib`allocateClassMemorySegment(javaVM=0x00007feb0580b820, requiredSize=<unavailable>, segmentType=65536, classLoader=0x00007feb068532e8, allocationIncrement=<unavailable>) at classseg.c:119:18 [opt]
    frame #21: 0x000000000b6865b3 libj9vm29.dylib`internalCreateRAMClassFromROMClassImpl(J9VMThread*, J9ClassLoader*, J9ROMClass*, unsigned long, J9Class*, J9ROMMethod**, long, int, J9Class*, unsigned long, J9Class*, J9CreateRAMClassState*, J9ClassLoader*, J9Class*, J9Module*) at createramclass.cpp:3389:16 [opt]
    frame #22: 0x000000000b686183 libj9vm29.dylib`internalCreateRAMClassFromROMClassImpl(vmThread=0x0000000016f0cc00, classLoader=0x00007feb068532e8, romClass=<unavailable>, options=4097, elementClass=0x0000000000000000, methodRemapArray=0x00007feb06853370, entryIndex=-1, locationType=0, classBeingRedefined=0x0000000000000000, packageID=140647403434728, superclass=0x0000000016c6cb00, state=0x0000700009fdab30, hostClassLoader=0x00007feb068532e8, hostClass=0x0000000000000000, module=0x0000000000000000) at createramclass.cpp:2402 [opt]
    frame #23: 0x000000000b6843cc libj9vm29.dylib`::internalCreateRAMClassFromROMClass(vmThread=<unavailable>, classLoader=0x00007feb068532e8, romClass=<unavailable>, options=<unavailable>, elementClass=0x0000000000000000, protectionDomain=<unavailable>, methodRemapArray=0x0000000000000000, entryIndex=-1, locationType=0, classBeingRedefined=0x0000000000000000, hostClass=0x0000000000000000) at createramclass.cpp:3053:11 [opt]
    frame #24: 0x000000000b7b36b5 libj9vm29.dylib`internalDefineClass(vmThread=0x0000000016f0cc00, className=0x0000700009fdae10, classNameLength=10, classData=<unavailable>, classDataLength=<unavailable>, classDataObject=<unavailable>, classLoader=<unavailable>, protectionDomain=0x00000000ffd48518, options=4097, existingROMClass=0x0000000000000000, hostClass=0x0000000000000000, localBuffer=0x0000700009fdad70) at defineclass.c:166:12 [opt]
    frame #25: 0x000000000d4aa575 libjclse29.dylib`defineClassCommon(env=0x0000000016f0cc00, classLoaderObject=0x00007feb068532e8, className=0x000000001700e2c8, classRep=0x000000001700e2c0, offset=<unavailable>, length=<unavailable>, protectionDomain=0x000000001700e288, options=0, hostClass=0x0000000000000000) at jcldefine.c:192:10 [opt]
    frame #26: 0x000000000d49cbc3 libjclse29.dylib`::Java_java_lang_ClassLoader_defineClassImpl(env=0x0000000016f0cc00, receiver=<unavailable>, className=0x000000001700e2c8, classRep=<unavailable>, offset=<unavailable>, length=<unavailable>, protectionDomain=0x000000001700e288) at clsldr.cpp:73:18 [opt]
    frame #27: 0x0000000023651046
    frame #28: 0x000000000b678101 libj9vm29.dylib`::sendLoadClass(currentThread=0x00000000ffd48560, classLoaderObject=0x00000000ffe9ea28, classNameObject=0x00000000ffe9cb28) at callin.cpp:440:3 [opt]
    frame #29: 0x000000000b67f43f libj9vm29.dylib`internalFindClassInModule at classsupport.c:652:3 [opt]
    frame #30: 0x000000000b67f429 libj9vm29.dylib`internalFindClassInModule [inlined] arbitratedLoadClass(vmThread=<unavailable>, className=<unavailable>, classNameLength=10, classLoader=<unavailable>, classNotFoundException=<unavailable>) at classsupport.c:853 [opt]
    frame #31: 0x000000000b67f429 libj9vm29.dylib`internalFindClassInModule [inlined] loadNonArrayClass(vmThread=<unavailable>, j9module=0x0000000000000000, className=<unavailable>, classNameLength=10, classLoader=<unavailable>, options=1, exception=<unavailable>) at classsupport.c:1044 [opt]
    frame #32: 0x000000000b67ef01 libj9vm29.dylib`internalFindClassInModule(vmThread=<unavailable>, j9module=0x0000000000000000, className=<unavailable>, classNameLength=10, classLoader=<unavailable>, options=1) at classsupport.c:1089 [opt]
    frame #33: 0x000000000b6b8978 libj9vm29.dylib`::resolveClassRef(vmStruct=<unavailable>, ramCP=0x0000000016f0a900, cpIndex=90, resolveFlags=65) at resolvesupport.cpp:303:18 [opt]
    frame #34: 0x000000000b62e0ec libj9vm29.dylib`VM_BytecodeInterpreter::run(J9VMThread*) [inlined] VM_BytecodeInterpreter::newLogic(unsigned long*&, unsigned char*&, J9Object**) at BytecodeInterpreter.hpp:7383:3 [opt]
    frame #35: 0x000000000b62e040 libj9vm29.dylib`VM_BytecodeInterpreter::run(J9VMThread*) [inlined] VM_BytecodeInterpreter::newdup(unsigned long*&, unsigned char*&) at BytecodeInterpreter.hpp:7416 [opt]
    frame #36: 0x000000000b62e040 libj9vm29.dylib`VM_BytecodeInterpreter::run(this=0x0000700009fdb5c8, vmThread=<unavailable>) at BytecodeInterpreter.hpp:10136 [opt]
    frame #37: 0x000000000b6102ac libj9vm29.dylib`::bytecodeLoop(currentThread=<unavailable>) at BytecodeInterpreter.cpp:109:21 [opt]
    frame #38: 0x000000000b750052 libj9vm29.dylib`cInterpreter + 22
    frame #39: 0x000000000b67bb6f libj9vm29.dylib`::sidecarInvokeReflectMethodImpl(currentThread=0x0000000016f8bec8, methodRef=<unavailable>, recevierRef=<unavailable>, argsRef=0x000000001700e840) at callin.cpp:1139:4 [opt]
    frame #40: 0x000000000b67c5a5 libj9vm29.dylib`::sidecarInvokeReflectMethod(currentThread=0x0000000016f0cc00, methodRef=0x000000001700e830, recevierRef=0x000000001700e848, argsRef=0x000000000a563108) at callin.cpp:1283:2 [opt]
    frame #41: 0x000000000d4dc67d libjclse29.dylib`JVM_InvokeMethod_Impl(env=0x0000000016f0cc00, method=0x000000001700e830, obj=0x000000001700e848, args=0x000000001700e840) at sunvmi.c:354:2 [opt]
    frame #42: 0x0000000023661aa9
    frame #43: 0x000000000b67bb6f libj9vm29.dylib`::sidecarInvokeReflectMethodImpl(currentThread=0x00000000ffe93e78, methodRef=<unavailable>, recevierRef=<unavailable>, argsRef=0x0000000016f10570) at callin.cpp:1139:4 [opt]
    frame #44: 0x000000000b67c5a5 libj9vm29.dylib`::sidecarInvokeReflectMethod(currentThread=0x0000000016f0cc00, methodRef=0x0000000016f10560, recevierRef=0x0000000000000000, argsRef=0x000000000a563108) at callin.cpp:1283:2 [opt]
    frame #45: 0x000000000d4dc67d libjclse29.dylib`JVM_InvokeMethod_Impl(env=0x0000000016f0cc00, method=0x0000000016f10560, obj=0x0000000000000000, args=0x0000000016f10570) at sunvmi.c:354:2 [opt]
    frame #46: 0x0000000023661aa9
    frame #47: 0x000000000b6790e2 libj9vm29.dylib`::runJavaThread(currentThread=0x00007feb03e0f700) at callin.cpp:645:4 [opt]
    frame #48: 0x000000000b746367 libj9vm29.dylib`javaProtectedThreadProc(portLibrary=<unavailable>, entryarg=0x0000000016f0cc00) at vmthread.c:2070:4 [opt]
    frame #49: 0x000000000b87faad libj9prt29.dylib`omrsig_protect(portLibrary=0x000000000a562348, fn=<unavailable>, fn_arg=0x0000000000000000, handler=<unavailable>, handler_arg=0x0000000016f0cc00, flags=<unavailable>, result=0x0000700009fdbed0) at omrsignal.c:384:12

Note: this crash occurs quite frequently and is easy to reproduce (in variable places though). However the core files just disappear when current test (not grinder) finishes. I have to use Ctrl-C to stop the test when the console just starts printing something like JVM requested System dump using '/ivm/macos/jdk11-test/jvmtest/openjdk_regression/work/scratch/4/core.20190410.171936.59665.0001.dmp' in response to an event.
So it appears the test script was pretty quick to do the cleanup, maybe this was the reason why the core files weren't found in grinders for #4778 or others.

fyi @smlambert @llxia

@smlambert
Copy link
Contributor

We do not yet capture core files from openjdk tests (dependent on this WIP PR adoptium/aqa-tests#1007), but thanks for letting me know of this failure that is repeatable, as I can use it to test my changes in that PR before merging.

@JasonFengJ9 JasonFengJ9 mentioned this issue Apr 11, 2019
Closed
@JasonFengJ9
Copy link
Member

@smlambert good to hear that adoptium/aqa-tests#1007 is going to archive the diagnostic files.
Just to clarify, the problem reported at #5399 (comment) is a local test run which seems not keep diagnostic files. I have created #5454 instead.

@dmitripivkine
Copy link
Contributor

I looked to one of crashes. It crashed in Global GC an attempt to scan slot of String Table in Symbol=_ZN27MM_MarkingSchemeRootClearer17doStringTableSlotEPP8J9ObjectP20GC_HashTableIterator at InaccessibleAddress=0000000100000110

There are eight hash tables in this String Table array. It crashed an attempt to scan very first slot for table

> !J9HashTable 0x00007f9566f10090
J9HashTable at 0x7f9566f10090 {
  Fields for J9HashTable:
	0x0: const U8* tableName = !j9x 0x000000000958142F
	0x8: U32 tableSize = 0x00000493 (1171)
	0xc: U32 numberOfNodes = 0x0000032C (812)
	0x10: U32 numberOfTreeNodes = 0x00000000 (0)
	0x14: U32 entrySize = 0x00000008 (8)
	0x18: U32 listNodeSize = 0x00000010 (16)
	0x1c: U32 treeNodeSize = 0x00000018 (24)
	0x20: U32 nodeAlignment = 0x00000008 (8)
	0x24: U32 flags = 0x00000002 (2)
	0x28: U32 memoryCategory = 0x80000003 (2147483651)
	0x2c: U32 listToTreeThreshold = 0x00000400 (1024)
	0x30: void** nodes = !j9x 0x00007F956904EA20 <------------------------------
	0x38: struct J9Pool* listNodePool = !j9pool 0x00007F9566F10160
	0x40: struct J9Pool* treeNodePool = !j9pool 0x00007F9566F103C0
	0x48: struct J9Pool* treePool = !j9pool 0x00007F9566F10250
	0x50: struct J9AVLTree* avlTreeTemplate = !j9avltree 0x00007F9566F10340
	0x58: void* hashFn = !j9x 0x00000000093DE440
	0x60: void* hashEqualFn = !j9x 0x00000000094BEDA0
	0x68: void* printFn = !j9x 0x0000000000000000
	0x70: struct OMRPortLibrary* portLibrary = !omrportlibrary 0x0000000007F41C28
	0x78: void* equalFnUserData = !j9x 0x00007F9566F10340
	0x80: void* hashFnUserData = !j9x 0x00007F9567804620
	0x88: struct J9HashTable* previous = !j9hashtable 0x0000000000000000
}

because of bad pointer (or memory corrupted content) for nodes = !j9x 0x00007F956904EA20:

0x7F956904EA20 :  0000000100000110 0000000000000000 [ ................ ]

This hash table is not converted to AVL tree so an address at 0x7F956904EA20 should point to one of puddles for listNodePool = !j9pool 0x00007F9566F10160 (use !walkj9pool 0x00007F9566F10160 for reference) but this is inaccessible address instead.

Please contact me if you need more information or preserved system core(s)

@JasonFengJ9
Copy link
Member

JasonFengJ9 commented May 21, 2019 •
edited
Loading

Will pick up this.

Was on other items instead.

@JasonFengJ9 JasonFengJ9 mentioned this issue Jun 3, 2019
Closed
@smlambert smlambert added the AQA Test failures from AdoptOpenJDK Quality Assurance test set label Jun 14, 2019
@pshipton
Copy link
Member

It doesn't look like we'll have a solution ready for this in the 0.15 release.

@adamfarley
Copy link
Contributor Author

adamfarley commented Aug 12, 2019 •
edited
Loading

Re: #5399 (comment)

Hey @dmitripivkine, looks like a user in the OpenJ9 slack is seeing a problem that looks like this. Especially this bit:

_ZN22VM_BytecodeInterpreter3runEP10J9VMThread

https://openj9.slack.com/archives/C8312LCV9/p1565452794048200

@adamfarley
Copy link
Contributor Author

Could this be related to this issue's original cause, or should I open a new issue to cover it?

@smlambert
Copy link
Contributor

FYI, Dmitri is away for the week.

@dmitripivkine
Copy link
Contributor

@adamfarley Would you try to reproduce the problem with latest JVM? DDR is fixed now, so we can investigate system core

@dmitripivkine
Copy link
Contributor

dmitripivkine commented Aug 27, 2019 •
edited
Loading

Looking to information dumped before. Looks like most likely this is the same problem (the same pattern) as #5988
so @adamfarley it makes sense to wait for fix before re-spin

@theresa-m theresa-m mentioned this issue Sep 17, 2019
Merged
@pshipton
Copy link
Member

This should now be fixed, if there are tests excluded they can be unexcluded.

@adamfarley
Copy link
Contributor Author

To my knowledge, no tests were excluded as part of this bug. No tests will be reintroduced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
AQA Test failures from AdoptOpenJDK Quality Assurance test set comp:vm test failure
Projects
None yet
Milestone
Release 0.17 (Java 8, 11, 13) Oct ref...
Development

Successfully merging a pull request may close this issue.

Round maxRequiredSize to match romSize calculation theresa-m/openj9
6 participants
@smlambert @DanHeidinga @JasonFengJ9 @pshipton @dmitripivkine @adamfarley