cmdLineTester_criu_jitserverPostRestore Unable to create a thread

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_s390x_linux_Nightly_testList_1/56 - ub20-390-3a
cmdLineTester_criu_jitserverPostRestore_1

Testing: Test SSL Failure Case with mismatched certificate
Test start time: 2023/09/13 23:49:02 Coordinated Universal Time
Running command: bash /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Nightly_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Nightly_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/bin " -Xjit:count=0 " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -XX:JITServerSSLRootCerts=wrongCert.pem -Xjit:verbose={compilePerformance},verbose={CheckpointRestore},verbose={JITServer},verbose={JITServerConns},vlog=sslVlog2" 1 false true
Time spent starting: 1 milliseconds
Time spent executing: 17057 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] Generate SSL certificates
 [OUT] Certificates generated
 [OUT] Starting /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=49458 -XX:JITServerSSLKey=key.pem -XX:JITServerSSLCert=cert.pem -Xjit:verbose={JITServer}
 [OUT] 2655879 ?        00:00:00 jitserver
 [OUT] JITSERVER EXISTS
 [OUT] Pre-checkpoint
 [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Wed Sep 13 23:49:13 UTC 2023, System.currentTimeMillis(): 1694648954304, System.nanoTime(): 1694648954301693211
 [OUT] pie: 2655958: Error (criu/pie/restorer.c:1833): Unable to create a thread: 2655961
 [OUT] pie: 2655958: Error (criu/pie/restorer.c:1968): Restorer fail 2655958
 [OUT] Error (criu/cr-restore.c:2536): Restoring FAILED.
 [OUT] 2655879 ?        00:00:00 jitserver
 [OUT] JITSERVER STILL EXISTS
 [OUT] Terminating /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=49458 -XX:JITServerSSLKey=key.pem -XX:JITServerSSLCert=cert.pem -Xjit:verbose={JITServer}
 [OUT] finished script
 [ERR] Generating RSA private key, 2048 bit long modulus (2 primes)
 [ERR] .+++++
 [ERR] ....................................+++++
 [ERR] e is 65537 (0x010001)
 [ERR] Generating a RSA private key
 [ERR] ..................+++++
 [ERR] ..................................................................................................................................................+++++
 [ERR] writing new private key to 'wrongKey.pem'
 [ERR] -----
 [ERR] JITServer is currently a technology preview. Its use is not yet supported.
 [ERR] #JITServer: JITServer version: 1.51.0
 [ERR] #JITServer: JITServer Server Mode. Port: 49458. Connection Timeout 30000ms
 [ERR] #JITServer: Built against (OpenSSL 3.0.11-dev ); Loaded with (OpenSSL 1.1.1f  31 Mar 2020)
 [ERR] 
 [ERR] #INFO:  StartTime: Sep 13 23:49:04 2023
 [ERR] #INFO:  TimeZone: UTC (UTC)
 [ERR] #JITServer: Started JITServer listener thread: 000000000021F800 
 [ERR] #JITServer: Successfully initialized SSL context (OpenSSL 1.1.1f  31 Mar 2020)
 [ERR] 
 [ERR] 
 [ERR] JITServer is ready to accept incoming requests
 [ERR] /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Nightly_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 113: 2655958 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
 [ERR] /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Nightly_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 113: 2655879 Killed                  $TEST_JDK_BIN/jitserver $JITSERVER_OPTIONS
>> Success condition was found: [Output match: Killed]
>> Required condition was found: [Output match: Pre-checkpoint]
>> Success condition was not found: [Output match: Post-checkpoint]
>> Failure condition was not found: [Output match: CRIU is not enabled]
>> Failure condition was not found: [Output match: Operation not permitted]
>> Required condition was found: [Output match: Successfully initialized SSL context]
>> Required condition was not found: [Output match: certificate verify failed]
>> Success condition was not found: [Output match: Thread pid mismatch]
>> Success condition was not found: [Output match: do not match expected]
>> Success condition was found: [Output match: Unable to create a thread:]
>> Success condition was found: [Output match: JITSERVER EXISTS]
>> Success condition was found: [Output match: JITSERVER STILL EXISTS]
>> Failure condition was not found: [Output match: JITSERVER DOES NOT EXIST]
>> Failure condition was not found: [Output match: JITSERVER NO LONGER EXISTS]

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_ppc64le_linux_aot_Personal_testList_1/194/ - ub16p8j93
cmdLineTester_criu_jitserverPostRestore_1

=== AOT ITERATION 2 ===

Testing: Test SSL Failure Case with mismatched certificate
Test start time: 2023/09/14 10:19:07 Eastern Standard Time
Running command: bash /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_ppc64le_linux_aot_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_ppc64le_linux_aot_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_ppc64le_linux_aot_Personal_testList_1/openjdkbinary/j2sdk-image/bin " -Xjit:count=0 " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -XX:JITServerSSLRootCerts=wrongCert.pem -Xjit:verbose={compilePerformance},verbose={CheckpointRestore},verbose={JITServer},verbose={JITServerConns},vlog=sslVlog2" 1 false true
Time spent starting: 3 milliseconds
Time spent executing: 8010 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] Generate SSL certificates
 [OUT] Certificates generated
 [OUT] Starting /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_ppc64le_linux_aot_Personal_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=41520 -XX:JITServerSSLKey=key.pem -XX:JITServerSSLCert=cert.pem -Xjit:verbose={JITServer}
 [OUT] 3450305 ?        00:00:00 jitserver
 [OUT] JITSERVER EXISTS
 [OUT] Pre-checkpoint
 [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Thu Sep 14 10:19:10 EDT 2023, System.currentTimeMillis(): 1694701150838, System.nanoTime(): 11662223735865687
 [OUT] pie: 3450382: Error (criu/pie/restorer.c:1833): Unable to create a thread: 3450384
 [OUT] pie: 3450384: Error (criu/pie/restorer.c:598): Thread pid mismatch 3450384/3450383
 [OUT] pie: 3450384: Error (criu/pie/restorer.c:651): Restorer abnormal termination for 3450382
 [OUT] Error (criu/cr-restore.c:2536): Restoring FAILED.
 [OUT] Error (criu/cr-restore.c:1494): 3450382 exited, status=1
 [OUT] 3450305 ?        00:00:00 jitserver
 [OUT] JITSERVER STILL EXISTS
 [OUT] Terminating /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_ppc64le_linux_aot_Personal_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=41520 -XX:JITServerSSLKey=key.pem -XX:JITServerSSLCert=cert.pem -Xjit:verbose={JITServer}
 [OUT] finished script
 [ERR] Generating RSA private key, 2048 bit long modulus (2 primes)
 [ERR] .........................+++++
 [ERR] .................................+++++
 [ERR] e is 65537 (0x010001)
 [ERR] Generating a RSA private key
 [ERR] .................+++++
 [ERR] ...........................................+++++
 [ERR] writing new private key to 'wrongKey.pem'
 [ERR] -----
 [ERR] #JITServer: JITServer version: 1.51.0
 [ERR] #JITServer: JITServer Server Mode. Port: 41520. Connection Timeout 30000ms
 [ERR] #JITServer: Built against (OpenSSL 3.0.11-dev ); Loaded with (OpenSSL 1.1.1f  31 Mar 2020)
 [ERR] 
 [ERR] #JITServer: Started JITServer listener thread: 00000000001CAF00 
 [ERR] #INFO:  StartTime: Sep 14 10:19:07 2023
 [ERR] #INFO:  TimeZone: UTC-4 (EDT)
 [ERR] 
 [ERR] JITServer is ready to accept incoming requests
 [ERR] #JITServer: Successfully initialized SSL context (OpenSSL 1.1.1f  31 Mar 2020)
 [ERR] 
 [ERR] /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_ppc64le_linux_aot_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 113: 3450382 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
 [ERR] /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_ppc64le_linux_aot_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 113: 3450305 Killed                  $TEST_JDK_BIN/jitserver $JITSERVER_OPTIONS
>> Success condition was found: [Output match: Killed]
>> Required condition was found: [Output match: Pre-checkpoint]
>> Success condition was not found: [Output match: Post-checkpoint]
>> Failure condition was not found: [Output match: CRIU is not enabled]
>> Failure condition was not found: [Output match: Operation not permitted]
>> Required condition was found: [Output match: Successfully initialized SSL context]
>> Required condition was not found: [Output match: certificate verify failed]
>> Success condition was found: [Output match: Thread pid mismatch]
>> Success condition was not found: [Output match: do not match expected]
>> Success condition was found: [Output match: Unable to create a thread:]
>> Success condition was found: [Output match: JITSERVER EXISTS]
>> Success condition was found: [Output match: JITSERVER STILL EXISTS]
>> Failure condition was not found: [Output match: JITSERVER DOES NOT EXIST]
>> Failure condition was not found: [Output match: JITSERVER NO LONGER EXISTS]

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_s390x_linux_aot_Personal_testList_1/198
cmdLineTester_criu_jitserverPostRestore_2

=== AOT ITERATION 1 ===

Testing: Check SSL Verbose Log for connection failure with Non SSL Server
Test start time: 2023/09/14 17:12:22 Coordinated Universal Time
Running command: bash /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_aot_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuCatVlog.sh sslVlog3 true true
Time spent starting: 2 milliseconds
Time spent executing: 11 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] vlog sslVlog3 does not exist
 [OUT] 
 [OUT] Outputting previous test output
 [OUT] Pre-checkpoint
 [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Thu Sep 14 17:12:15 UTC 2023, System.currentTimeMillis(): 1694711535368, System.nanoTime(): 1694711535365637347
 [OUT] pie: 2079969: Error (criu/pie/restorer.c:1833): Unable to create a thread: 2079971
 [OUT] pie: 2079969: Error (criu/pie/restorer.c:1968): Restorer fail 2079969
 [OUT] Error (criu/cr-restore.c:2536): Restoring FAILED.
 [OUT] 
 [OUT] CAT VLOG FORCE PASS
 [OUT] Removed test output files
 [OUT] finished script
>> Success condition was not found: [Output match: CHECKPOINT RESTORE: Ready for restore]
>> Success condition was not found: [Output match: JITServer::StreamFailure: Failed to SSL_connect]
>> Required condition was not found: [Output match: Could not connect to a server]
>> Success condition was found: [Output match: CAT VLOG FORCE PASS]

[pshipton]

@TobiAjila

[dsouzai]

This seems to fail because of the Required condition, which will never succeed if we don't have a vlog (which is the point of the CAT VLOG FORCE PASS output). @SajinaKandy I think you may need to alter the required condition so that known failures (such as Unable to create a thread) silently succeed.