Failed to execute cosbench work under multiple drivers per client:cannot load driver configuration

[HonryZhang]

Env info：
4 clients and start 3 drivers per client, total 12 clients
4 RGW nodes and start 3 radosgw process per node with port 7480-7482

config cosbench workload file: each driver connect to 3 rgw endporing port, as shown below, driver1 connect to 7480-7482 of rgw node1, driver2 connect to 7480-7482 of rgw node2... so there have 36 workers in the main workstage. each work put objects with different prefix.

<workstage name="main"> <work driver="driver1" name="1xxxx_7480_0" runtime="7200" totalOps="10000" workers="25"> <storage config="accesskey=xxxx;secretkey=xxxx;endpoint=http://xxx.76.34:7480; path_style_access=true;timeout=180000;" type="s3"/> <operation config="cprefix=i9run1bwbucket-;containers=r(1,1);oprefix=I9RuN1bW_34_7480_0_;objects=s(1,10000);sizes=c(4194304)B;hashCheck=False" ratio="100" type="write"/> </work> <work driver="driver1" name="xxx_7481_0" runtime="7200" totalOps="10000" workers="25"> <storage config="accesskey=xxxx;secretkey=xxxx;endpoint=http://xxx.76.34:7481; path_style_access=true;timeout=180000;" type="s3"/> <operation config="cprefix=i9run1bwbucket-;containers=r(1,1);oprefix=I9RuN1bW_34_7481_0_;objects=s(1,10000);sizes=c(4194304)B;hashCheck=False" ratio="100" type="write"/> </work> <work driver="driver1" name="1xxx_7482_0" runtime="7200" totalOps="10000" workers="25"> <storage config="accesskey=xxxx;secretkey=xxxx;endpoint=http://1xxx76.34:7482; path_style_access=true;timeout=180000;" type="s3"/> <operation config="cprefix=i9run1bwbucket-;containers=r(1,1);oprefix=I9RuN1bW_34_7482_0_;objects=s(1,10000);sizes=c(4194304)B;hashCheck=False" ratio="100" type="write"/> </work> <work driver="driver2" name="1xxx._7480_1" runtime="7200" totalOps="10000" workers="25"> <storage config="accesskey=xxxx;secretkey=xxxx;endpoint=http://xxx.76.35:7480; path_style_access=true;timeout=180000;" type="s3"/> <operation config="cprefix=i9run1bwbucket-;containers=r(1,1);oprefix=I9RuN1bW_35_7480_1_;objects=s(1,10000);sizes=c(4194304)B;hashCheck=False" ratio="100" type="write"/> </work> <work driver="driver2" name="xxx_7481_1" runtime="7200" totalOps="10000" workers="25"> <storage config="accesskey=xxxx;secretkey=xxxx;endpoint=http://xxx.76.35:7481; path_style_access=true;timeout=180000;" type="s3"/> <operation config="cprefix=i9run1bwbucket-;containers=r(1,1);oprefix=I9RuN1bW_35_7481_1_;objects=s(1,10000);sizes=c(4194304)B;hashCheck=False" ratio="100" type="write"/> </work> <work driver="driver2" name="1xxx_7482_1" runtime="7200" totalOps="10000" workers="25"> <storage config="accesskey=xxxx;secretkey=xxxx;endpoint=http://xxx.76.35:7482; path_style_access=true;timeout=180000;" type="s3"/> <operation config="cprefix=i9run1bw-ucket-;containers=r(1,1);oprefix=I9RuN1bW_35_7482_1_;objects=s(1,10000);sizes=c(4194304)B;hashCheck=False" ratio="100" type="write"/> </work>

start all drivers successfully and state are alive, submit the workload file but work stuck at the main workstage, check the detail info on web and find the state of driver3 (client3)is created. the others are under booted state. A few mins later, work terminated.

find below error logs in libs.log on client3:
2023-12-28 16:09:41,413 [ERROR] [SpringOsgiExtenderThread-22] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor - Expecting state (RESOLVING_DEPENDENCIES) not (DEPENDENCIES_RESOLVED) for context [OsgiBundleXmlApplicationContext(bundle=cosbench-driver, config=osgibundle:/META-INF/spring/.xml)]; assuming an interruption and bailing out
2023-12-28 16:09:41,562 [ERROR] [SpringOsgiExtenderThread-24] org.springframework.osgi.extender.internal.support.DefaultOsgiBundleApplicationContextListener - Application context refresh failed (OsgiBundleXmlApplicationContext(bundle=cosbench-driver, config=osgibundle:/META-INF/spring/.xml))
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'factory' defined in URL [bundleentry://79.fwk1689843956/META-INF/spring/driver-context.xml]: Invocation of init method failed; nested exception is com.intel.cosbench.config.ConfigException: cannot load driver configuration
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1420)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:62)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:327)
at org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:87)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:292)
at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:138)
at java.lang.Thread.run(Thread.java:750)
Caused by: com.intel.cosbench.config.ConfigException: cannot load driver configuration
at com.intel.cosbench.service.AbstractServiceFactory.loadConfig(AbstractServiceFactory.java:62)
at com.intel.cosbench.service.AbstractServiceFactory.init(AbstractServiceFactory.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1544)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
... 14 more
Caused by: com.intel.cosbench.config.ConfigException: cannot loader config from /home/Cosbench/conf/driver.conf
at com.intel.cosbench.config.common.INIConfigParser.parse(INIConfigParser.java:45)
at com.intel.cosbench.service.AbstractServiceFactory.loadConfig(AbstractServiceFactory.java:59)
... 22 more
Caused by: org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source driver.conf
at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:215)
at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:195)
at org.apache.commons.configuration.AbstractFileConfiguration.(AbstractFileConfiguration.java:160)
at org.apache.commons.configuration.INIConfiguration.(INIConfiguration.java:209)
at com.intel.cosbench.config.common.INIConfigParser.parse(INIConfigParser.java:41)
... 23 more

confirmed that driver.conf file exist under the path. I also tried to stop the start all drivers, but still encounter the same issus occasionally. Perhaps driver5 or driver8 will under booted state at next time.

Any idea about this issue?