Dashboard returns 500 - Cannot serve resource

[isaacgale]

What happened?

Running 3.5.16, after Reposilite has been running for a few days the UI become unresponsive and returns 500 errors. Ivy and Maven will still resolve artifacts for the most part. I don't see any other output in my current logging config to help identify the cause.

Reposilite version

3.x

Relevant log output

ERR RESULT | GET / errored with ErrorResponse(status=500, message=Cannot serve resource)

[dzikoysk]

It's probably some fs-related issue, like too many locked up resources or exhausted file handles. It's not coincidence that this started happening recently, so let's start with better logging.

[dzikoysk]

I quickly ran a few requests within the 2min window, using old & new endpoints, but unfortunately I didn't notice anything weird:

failures
21:47:19.382 INFO | 
21:47:19.382 INFO | No exception has occurred yet
21:47:19.382 INFO | 

It'd be great if you could move to the latest nightly, so we can try to catch your IOException.

[isaacgale]

The server had been running for 5ish days with a decent load on it (it serves a Jenkins server that resolves and publishes to the repository).

I've set debug to true in the configuration file for now, hopefully that will show some more information. I am running Reposilite on the same server I was previously running Nexus v2 on. Looking at the output of lsof for the Reposilite process it didn't seem to have an excessive number of files open. Also the ulimit for the server wasn't being reached either. I'll add more information as I can, are there any other logging settings I should change?

Like I said initially it only seemed to be affecting the dashboard. Builds were still able to resolve and publish in this state.

[dzikoysk]

I'll add more information as I can, are there any other logging settings I should change?

I think it's fine to keep it as it was, just remember to use the latest nightly. Assuming that we will eventually fall into the same problem, you should be able to see an exception being thrown when this error message is returned:

ERR RESULT | GET / errored with ErrorResponse(status=500, message=Cannot serve resource)

It was just a matter of a poor implementation that this has been lost. Having the exception, it'll be a bit easier to look for a root cause 🤞

[isaacgale]

I will look at updating to the latest snapshot then, sorry I misunderstood why you said to do that initially.

[isaacgale]

Ok this is the issue. I've discovered this server has a cronjob that is cleaning 'old' directories from /tmp which is causing this. I'm going to repoint the Java temp dir to a different location for now.

03:07:03.800 ERROR | index.html
03:07:03.800 ERROR | java.nio.file.NoSuchFileException: /tmp/reposilite6646411961449879734frontend-resource
03:07:03.800 ERROR | at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
03:07:03.800 ERROR | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
03:07:03.800 ERROR | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
03:07:03.800 ERROR | at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
03:07:03.800 ERROR | at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
03:07:03.801 ERROR | at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
03:07:03.801 ERROR | at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
03:07:03.801 ERROR | at java.base/java.nio.file.Files.newInputStream(Files.java:160)
03:07:03.801 ERROR | at com.reposilite.frontend.LazyPlaceholderResolver.createProcessedResource$lambda$4$lambda$3(LazyPlaceholderResolver.kt:51)
03:07:03.801 ERROR | at panda.std.Result.supplyThrowing(Result.java:133)
03:07:03.801 ERROR | at com.reposilite.frontend.LazyPlaceholderResolver.createProcessedResource$lambda$4(LazyPlaceholderResolver.kt:50)
03:07:03.801 ERROR | at com.reposilite.frontend.infrastructure.FrontendHandler.respondWithProcessedResource(FrontendHandler.kt:63)
03:07:03.801 ERROR | at com.reposilite.frontend.infrastructure.FrontendHandler.respondWithResource(FrontendHandler.kt:54)
03:07:03.801 ERROR | at com.reposilite.frontend.infrastructure.ResourcesFrontendHandler.respondWithBundledResource(FrontendHandler.kt:96)
03:07:03.801 ERROR | at com.reposilite.frontend.infrastructure.ResourcesFrontendHandler.access$respondWithBundledResource(FrontendHandler.kt:77)
03:07:03.801 ERROR | at com.reposilite.frontend.infrastructure.ResourcesFrontendHandler$defaultHandler$1.invoke(FrontendHandler.kt:84)
03:07:03.801 ERROR | at com.reposilite.frontend.infrastructure.ResourcesFrontendHandler$defaultHandler$1.invoke(FrontendHandler.kt:83)
03:07:03.801 ERROR | at com.reposilite.web.infrastructure.ReposiliteRoutingKt$createReposiliteDsl$dsl$1.invoke$lambda$0(ReposiliteRouting.kt:78)
03:07:03.801 ERROR | at io.javalin.router.Endpoint.handle(Endpoint.kt:52)
03:07:03.801 ERROR | at io.javalin.router.ParsedEndpoint.handle(ParsedEndpoint.kt:15)
03:07:03.801 ERROR | at io.javalin.http.servlet.DefaultTasks.HTTP$lambda$9$lambda$7$lambda$6(DefaultTasks.kt:52)
03:07:03.801 ERROR | at io.javalin.http.servlet.JavalinServlet.handleTask(JavalinServlet.kt:99)
03:07:03.801 ERROR | at io.javalin.http.servlet.JavalinServlet.handleSync(JavalinServlet.kt:64)
03:07:03.801 ERROR | at io.javalin.http.servlet.JavalinServlet.handle(JavalinServlet.kt:50)
03:07:03.801 ERROR | at io.javalin.http.servlet.JavalinServlet.service(JavalinServlet.kt:30)
03:07:03.801 ERROR | at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587)
03:07:03.801 ERROR | at io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:52)
03:07:03.801 ERROR | at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587)
03:07:03.801 ERROR | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
03:07:03.801 ERROR | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:529)
03:07:03.801 ERROR | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
03:07:03.801 ERROR | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
03:07:03.801 ERROR | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
03:07:03.801 ERROR | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
03:07:03.801 ERROR | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
03:07:03.801 ERROR | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
03:07:03.801 ERROR | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
03:07:03.801 ERROR | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
03:07:03.801 ERROR | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
03:07:03.801 ERROR | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
03:07:03.801 ERROR | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
03:07:03.801 ERROR | at org.eclipse.jetty.server.Server.handle(Server.java:563)
03:07:03.801 ERROR | at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
03:07:03.802 ERROR | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
03:07:03.802 ERROR | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
03:07:03.802 ERROR | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
03:07:03.802 ERROR | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
03:07:03.802 ERROR | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
03:07:03.802 ERROR | at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
03:07:03.802 ERROR | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
03:07:03.802 ERROR | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
03:07:03.802 ERROR | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
03:07:03.802 ERROR | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
03:07:03.802 ERROR | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
03:07:03.802 ERROR | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
03:07:03.802 ERROR | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
03:07:03.802 ERROR | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
03:07:03.802 ERROR | at java.base/java.lang.Thread.run(Thread.java:833)

[dzikoysk]

Okay, I'm glad we've found the issue. I'd say that's not a bug then. It is not expected that tmp directory is cleaned up when the system is running - it should be done when you're booting the OS and the /tmp is being mounted.

[JasperTeng]

@dzikoysk I do not have a cronjob cleaning up /tmp and I am facing this issue constantly after a period of idle. Is there a way to specify a different location for this file? or perhaps can reposilite lock the file when it is running, to prevent it from being removed by OS due to it being stale?

[JasperTeng]

@dzikoysk I do not have a cronjob cleaning up /tmp and I am facing this issue constantly after a period of idle. Is there a way to specify a different location for this file? or perhaps can reposilite lock the file when it is running, to prevent it from being removed by OS due to it being stale?

I meant i do not have a additional cronjob that i created myself.
cleanup is standard from OS out of the box (e.g.: /etc/tmpfiles.d/tmp.conf)

[dzikoysk]

@JasperTeng You can change temp directory used by JVM:

$ java -Djava.io.tmpdir=/path/to/tmpdir