FISH-9222 bugfix: disable PayaraExecutorService events before server shuts down

[lprimak]

Description

Disable PayaraExecutorService events before server shuts down.
Prevents exceptions on shutdown as described in linked issue
fixes #6879

[lprimak]

@Pandrex247 Another one for you :)

[lprimak]

The fix is incomplete. Got another exception:
Fixed now:

[2024-08-17T22:19:50.693-0500] [Payara 6.2024.9] [WARNING] [] [fish.payara.appserver.micro.services.PayaraInstanceImpl] [tid: _ThreadID=137 _ThreadName=payara-executor-service-scheduled-task] [timeMillis: 1723951190693] [levelValue: 900] [[
  Failed to get running Grizzly listener.
MultiException stack 1 of 1
java.lang.IllegalStateException: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 1 but it has a run level of 10.  The full descriptor is SystemDescriptor(
	implementation=com.sun.enterprise.v3.services.impl.GrizzlyService
	contracts={com.sun.enterprise.v3.services.impl.GrizzlyService,org.glassfish.api.container.RequestDispatcher,fish.payara.monitoring.collect.MonitoringDataSource}
	scope=org.glassfish.hk2.runlevel.RunLevel
	qualifiers={}
	descriptorType=CLASS
	descriptorVisibility=NORMAL
	metadata=runLevelValue={10},Bundle-SymbolicName={fish.payara.server.core.nucleus.kernel},Bundle-Version={6.19.0.SNAPSHOT}
	rank=50
	loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [fish.payara.server.core.nucleus.kernel [322]], State = [READY],1281793904)
	proxiable=null
	proxyForSameScope=null
	analysisName=null
	id=1160
	locatorId=0
	identityHashCode=449083851
	reified=true)
	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.validate(AsyncRunLevelContext.java:422)
	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:275)
	at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2103)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:769)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:732)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:702)
	at fish.payara.appserver.micro.services.PayaraInstanceImpl.initialiseInstanceDescriptor(PayaraInstanceImpl.java:407)
	at org.glassfish.grizzly.utils.Holder$3.evaluate(Holder.java:50)
	at org.glassfish.grizzly.utils.Holder$LazyHolder.get(Holder.java:85)
	at fish.payara.appserver.micro.services.PayaraInstanceImpl.lambda$event$0(PayaraInstanceImpl.java:253)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1570)
]]

[breakponchito]

@lprimak I tried to replicate but no success until now, as you said Do I need to shutdown the server with the monitor console enable for around 30 times? If you have clear steps to replicate please share to me

[lprimak]

Sorry, this happens only on CI server. Only rarely. I have no reliable reproducer for you unfortunately, but it's pretty clear what happens here though.

[Pandrex247]

@breakponchito

[2024-08-17T22:19:50.693-0500] [Payara 6.2024.9] [WARNING] [] [fish.payara.appserver.micro.services.PayaraInstanceImpl] [tid: _ThreadID=137 _ThreadName=payara-executor-service-scheduled-task] [timeMillis: 1723951190693] [levelValue: 900] [[
  Failed to get running Grizzly listener.
MultiException stack 1 of 1
java.lang.IllegalStateException: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 1 but it has a run level of 10.  The full descriptor is SystemDescriptor(

I've managed to see that error on Windows.

• Start a DAS
• Create and start an instance
• Create a deployment group and add the instance to the deployment group
• From the admin console - click stop deployment group
• Check the logs of the instance

[breakponchito]

@lprimak @Pandrex247 I already replicated the issue on windows following the steps you shared @Pandrex247 thank you:
[2024-08-23T21:15:26.828-0600] [Payara 6.2024.9] [ADVERTENCIA] [] [fish.payara.appserver.micro.services.PayaraInstanceImpl] [tid: _ThreadID=166 _ThreadName=payara-executor-service-scheduled-task] [timeMillis: 1724469326828] [levelValue: 900] [[ Failed to get running Grizzly listener. MultiException stack 1 of 1 java.lang.IllegalStateException: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 1 but it has a run level of 10. The full descriptor is SystemDescriptor( implementation=com.sun.enterprise.v3.services.impl.GrizzlyService contracts={com.sun.enterprise.v3.services.impl.GrizzlyService,org.glassfish.api.container.RequestDispatcher,fish.payara.monitoring.collect.MonitoringDataSource} scope=org.glassfish.hk2.runlevel.RunLevel qualifiers={} descriptorType=CLASS descriptorVisibility=NORMAL

[breakponchito]

LGTM

[breakponchito]

yes with this fix now the threads are finishing correctly and logs are clean, without any issue after shutdown deployment groups

[breakponchito]

@lprimak thank you for your contribution, this was already merged