main io.trino.server.Server No service providers of type io.trino.spi.Plugin in the classpath

[rsadinen]

Installed Trino server 438 and configured hive.connector Verified all configs ,permissions, java, they all looks good but not able to launch the trino server

ERROR#

java.lang.IllegalStateException: No service providers of type io.trino.spi.Plugin in the classpath: [file:/var/trino/data/plugin/raptor-legacy/HdrHistogram-2.1.9.jar, file:/var/trino/data/plugin/raptor-legacy/aircompressor-0.25.jar, file:/var/trino/data/plugin/raptor-legacy/aopalliance-1.0.jar, file:/var/trino/data/plugin/raptor-legacy/bootstrap-240.jar, file:/var/trino/data/plugin/raptor-legacy/byte-buddy-1.14.11.jar, file:/var/trino/data/plugin/raptor-legacy/checker-qual-3.42.0.jar, file:/var/trino/data/plugin/raptor-legacy/classmate-1.5.1.jar, file:/var/trino/data/plugin/raptor-legacy/concurrent-240.jar, file:/var/trino/data/plugin/raptor-legacy/configuration-240.jar, file:/var/trino/data/plugin/raptor-legacy/error_prone_annotations-2.24.1.jar, file:/var/trino/data/plugin/raptor-legacy/failureaccess-1.0.2.jar, file:/var/trino/data/plugin/raptor-legacy/fastutil-8.5.12.jar, file:/var/trino/data/plugin/raptor-legacy/geantyref-1.3.15.jar, file:/var/trino/data/plugin/raptor-legacy/guava-33.0.0-jre.jar, file:/var/trino/data/plugin/raptor-legacy/guice-7.0.0.jar, file:/var/trino/data/plugin/raptor-legacy/h2-2.2.224.jar, file:/var/trino/data/plugin/raptor-legacy/hibernate-validator-8.0.1.Final.jar, file:/var/trino/data/plugin/raptor-legacy/http-client-240.jar, file:/var/trino/data/plugin/raptor-legacy/http2-client-11.0.20.jar, file:/var/trino/data/plugin/raptor-legacy/http2-common-11.0.20.jar, file:/var/trino/data/plugin/raptor-legacy/http2-hpack-11.0.20.jar, file:/var/trino/data/plugin/raptor-legacy/http2-http-client-transport-11.0.20.jar, file:/var/trino/data/plugin/raptor-legacy/j2objc-annotations-2.8.jar, file:/var/trino/data/plugin/raptor-legacy/jackson-core-2.16.1.jar, file:/var/trino/data/plugin/raptor-legacy/jackson-databind-2.16.1.jar, file:/var/trino/data/plugin/raptor-legacy/jackson-datatype-guava-2.16.1.jar, file:/var/trino/data/plugin/raptor-legacy/jackson-datatype-jdk8-2.16.1.jar, file:/var/trino/data/plugin/raptor-legacy/jackson-datatype-joda-2.16.1.jar, file:/var/trino/data/plugin/raptor-legacy/jackson-datatype-jsr310-2.16.1.jar, file:/var/trino/data/plugin/raptor-legacy/jackson-module-parameter-names-2.16.1.jar, file:/var/trino/data/plugin/raptor-legacy/jakarta.annotation-api-2.1.1.jar, file:/var/trino/data/plugin/raptor-legacy/jakarta.inject-api-2.0.1.jar, file:/var/trino/data/plugin/raptor-legacy/jakarta.validation-api-3.0.2.jar, file:/var/trino/data/plugin/raptor-legacy/jboss-logging-3.4.3.Final.jar, file:/var/trino/data/plugin/raptor-legacy/jcl-over-slf4j-2.0.11.jar, file:/var/trino/data/plugin/raptor-legacy/jdbi3-core-3.44.0.jar, file:/var/trino/data/plugin/raptor-legacy/jdbi3-sqlobject-3.44.0.jar, file:/var/trino/data/plugin/raptor-legacy/jetty-alpn-client-11.0.20.jar, file:/var/trino/data/plugin/raptor-legacy/jetty-alpn-java-client-11.0.20.jar, file:/var/trino/data/plugin/raptor-legacy/jetty-client-11.0.20.jar, file:/var/trino/data/plugin/raptor-legacy/jetty-http-11.0.20.jar, file:/var/trino/data/plugin/raptor-legacy/jetty-io-11.0.20.jar, file:/var/trino/data/plugin/raptor-legacy/jetty-util-11.0.20.jar, file:/var/trino/data/plugin/raptor-legacy/jmxutils-1.25.jar, file:/var/trino/data/plugin/raptor-legacy/joda-time-2.12.6.jar, file:/var/trino/data/plugin/raptor-legacy/json-240.jar, file:/var/trino/data/plugin/raptor-legacy/jsr305-3.0.2.jar, file:/var/trino/data/plugin/raptor-legacy/log-240.jar, file:/var/trino/data/plugin/raptor-legacy/log-manager-240.jar, file:/var/trino/data/plugin/raptor-legacy/log4j-api-2.17.1.jar, file:/var/trino/data/plugin/raptor-legacy/log4j-over-slf4j-2.0.11.jar, file:/var/trino/data/plugin/raptor-legacy/log4j-to-slf4j-2.17.1.jar, file:/var/trino/data/plugin/raptor-legacy/logback-core-1.4.14.jar, file:/var/trino/data/plugin/raptor-legacy/modernizer-maven-annotations-2.7.0.jar, file:/var/trino/data/plugin/raptor-legacy/mysql-connector-j-8.2.0.jar, file:/var/trino/data/plugin/raptor-legacy/node-240.jar, file:/var/trino/data/plugin/raptor-legacy/opentelemetry-semconv-1.21.0-alpha.jar, file:/var/trino/data/plugin/raptor-legacy/orc-protobuf-14.jar, file:/var/trino/data/plugin/raptor-legacy/protobuf-java-3.25.2.jar, file:/var/trino/data/plugin/raptor-legacy/security-240.jar, file:/var/trino/data/plugin/raptor-legacy/slf4j-api-2.0.11.jar, file:/var/trino/data/plugin/raptor-legacy/slf4j-jdk14-2.0.11.jar, file:/var/trino/data/plugin/raptor-legacy/stats-240.jar, file:/var/trino/data/plugin/raptor-legacy/trace-token-240.jar, file:/var/trino/data/plugin/raptor-legacy/trino-array-438.jar, file:/var/trino/data/plugin/raptor-legacy/trino-cache-438.jar, file:/var/trino/data/plugin/raptor-legacy/trino-filesystem-438.jar, file:/var/trino/data/plugin/raptor-legacy/trino-matching-438.jar, file:/var/trino/data/plugin/raptor-legacy/trino-memory-context-438.jar, file:/var/trino/data/plugin/raptor-legacy/trino-orc-438.jar, file:/var/trino/data/plugin/raptor-legacy/trino-plugin-toolkit-438.jar, file:/var/trino/data/plugin/raptor-legacy/trino-raptor-legacy-438-services.jar, file:/var/trino/data/plugin/raptor-legacy/trino-raptor-legacy-438.jar, file:/var/trino/data/plugin/raptor-legacy/units-1.10.jar]
at com.google.common.base.Preconditions.checkState(Preconditions.java:835)
at io.trino.server.PluginManager.loadPlugin(PluginManager.java:165)
at io.trino.server.PluginManager.loadPlugin(PluginManager.java:155)
at io.trino.server.ServerPluginsProvider.lambda$loadPlugins$1(ServerPluginsProvider.java:57)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:186)
at io.trino.util.Executors.executeUntilFailure(Executors.java:46)
at io.trino.server.ServerPluginsProvider.loadPlugins(ServerPluginsProvider.java:52)
at io.trino.server.PluginManager.loadPlugins(PluginManager.java:137)
at io.trino.server.Server.doStart(Server.java:141)
at io.trino.server.Server.lambda$start$0(Server.java:91)
at io.trino.$gen.Trino_438____20240215_220142_1.run(Unknown Source)
at io.trino.server.Server.start(Server.java:91)
at io.trino.server.TrinoServer.main(TrinoServer.java:38)

2024-02-15T22:01:50.998Z INFO Thread-79 io.airlift.bootstrap.LifeCycleManager JVM is shutting down, cleaning up

[mosabua]

Without more info we cant tell whats wrong. Try to start it without catalogs and otherwise simplified config. Also how you are starting it where? And what binary are you using.

[rsadinen]

Launching trino using /usr/local/trino-server-438/bin/launcher start.
I renamed hive.connector to diff name and started trino ,it worked fine without any issues
Below is the hive.properties file which was working in another environment

connector.name=hive
hive.metastore.uri=thrift://hivehahttp0.dev.cloudera.site:9083
hive.metastore.authentication.type=KERBEROS
hive.metastore.service.principal=hive/[email protected]
hive.metastore.client.principal=[email protected]
hive.metastore.client.keytab=/usr/local/trino-server-438/dev-user.keytab
hive.hdfs.authentication.type=KERBEROS
hive.hdfs.trino.principal=[email protected]
hive.hdfs.trino.keytab=/usr/local/trino-server-438/dev-user.keytab

hive.config.resources=/usr/local/trino-server-438/core-site.xml,/usr/local/trino-server-438/hdfs-site.xml

[mosabua]

what do you mean by "renamed the hive.connector to a diff name"?

[mosabua]

And since it worked in another environment .. what is different?

[rsadinen]

I meant to say i removed files in catalog folder as you suggested ,I do not see any diff its same configuration .

[hashhar]

check the ulimits and make sure they are at-least as much as mentioned at https://trino.io/docs/current/installation/deployment.html#linux-operating-system

This can happen if number of open file limits is breached since Trino needs to load all the JARs at runtime and there are quite a lot of them.

[rsadinen]

Below are the limits i have ,Even with below values I am still getting same error
cat /proc/sys/fs/file-max
13035606

/etc/security/limits.conf

• hard core 0
• hard nofile 131072
• hard nproc 128000
• • soft nofile 131072
• soft nproc 128000

[mosabua]

I would suggest to try a bare bones deployment on a separate machine and troubleshoot one step at a time to figure out the difference. The release definitely works so there is an issue with your environment.

[hashhar]

/etc/security/limits.conf there are system-wide limits. It doesn't mean the user as which Trino runs has the soft limits set to these values. You need to login as the user under which Trino will run and then check output from ulimits -a to make sure the soft limits are bumped up.

Also +1 to what Manfred said, try to start with the simplest possible deployment. e.g. no catalogs configured at all.

[aliamala]

hi @rsadinen
is this problem solved?
I have the same issue

[junbangma]

I've seen the same issue with Trino 448 & 450. My use case is to build a docker image, with Trino as the base image and include my custom connector. I'm trying to deploy the docker image to kubernetes.
The root cause for me is that user:trino doesn't have the access to the classpath, which is /lib/trino in the container.
The solution is to add these in the Dockerfile

USER root
RUN chown -R "trino:trino" -R /lib/trino
USER trino

I'm not sure what other paths in the container are inaccessible to user:trino. This could be a potential issue in the future.