Maven dependency resolution performance

[AB-xdev]

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitLab

Please tell us more about your question or problem

I likely found 2 improvements regarding Maven dependency resolution for Maven Central (https://repo1.maven.org/maven2/).
I noticed these problems after further inspecting logs due to #27621 / #27759 and finding out that a small Maven project with ~30 dependencies sent around 3k request to maven central:

1. Renovate queries index.html to lookup Maven releases (e.g. https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/index.html), however Maven Central hosts no such file and the requests always fail.
2. I noticed that Renovate (without cache) sends extreme amounts of HEAD request to Maven central to detect/check all existing releases. However Artifacts on Maven Central are usually never revoked and if this should ever happen it will likely also be visible inside the maven-metadata.xml file (that is used for enumerating all dependency versions).

I recommend the following changes for https://repo1.maven.org/maven2/ and they likely should be ship by default with renovate:

1. Do not use index.html for lookup
2. Do not send HEAD requests for further validation of each dependency version

Logs (if relevant)

Logs

[2024-03-07T08:55:33.659Z] DEBUG: http statistics (repository=redacted)
[2024-03-07T08:55:33.659Z]        "urls": {
[2024-03-07T08:55:33.660Z]          "https://api.github.com/graphql (POST,200)": 7,
...
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.0/testcontainers-bom-1.10.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.1/testcontainers-bom-1.10.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.2/testcontainers-bom-1.10.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.3/testcontainers-bom-1.10.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.4/testcontainers-bom-1.10.4.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.5/testcontainers-bom-1.10.5.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.6/testcontainers-bom-1.10.6.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.7/testcontainers-bom-1.10.7.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.11.0/testcontainers-bom-1.11.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.11.1/testcontainers-bom-1.11.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.11.2/testcontainers-bom-1.11.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.11.3/testcontainers-bom-1.11.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.11.4/testcontainers-bom-1.11.4.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.12.0/testcontainers-bom-1.12.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.12.1/testcontainers-bom-1.12.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.12.2/testcontainers-bom-1.12.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.12.3/testcontainers-bom-1.12.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.12.4/testcontainers-bom-1.12.4.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.12.5/testcontainers-bom-1.12.5.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.13.0/testcontainers-bom-1.13.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.14.0/testcontainers-bom-1.14.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.14.1/testcontainers-bom-1.14.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.14.2/testcontainers-bom-1.14.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.14.3/testcontainers-bom-1.14.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.15.0-rc1/testcontainers-bom-1.15.0-rc1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.15.0-rc2/testcontainers-bom-1.15.0-rc2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.15.0/testcontainers-bom-1.15.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.15.1/testcontainers-bom-1.15.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.15.2/testcontainers-bom-1.15.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.15.3/testcontainers-bom-1.15.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.16.0/testcontainers-bom-1.16.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.16.1/testcontainers-bom-1.16.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.16.2/testcontainers-bom-1.16.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.16.3/testcontainers-bom-1.16.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.0/testcontainers-bom-1.17.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.1/testcontainers-bom-1.17.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.2/testcontainers-bom-1.17.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.3/testcontainers-bom-1.17.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.4/testcontainers-bom-1.17.4.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.5/testcontainers-bom-1.17.5.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.6/testcontainers-bom-1.17.6.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.18.0/testcontainers-bom-1.18.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.18.1/testcontainers-bom-1.18.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.18.2/testcontainers-bom-1.18.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.18.3/testcontainers-bom-1.18.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.0/testcontainers-bom-1.19.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.1/testcontainers-bom-1.19.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.2/testcontainers-bom-1.19.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.3/testcontainers-bom-1.19.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.4/testcontainers-bom-1.19.4.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.5/testcontainers-bom-1.19.5.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.6/testcontainers-bom-1.19.6.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.7/testcontainers-bom-1.19.7.pom (GET,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.7/testcontainers-bom-1.19.7.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.7.0/testcontainers-bom-1.7.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.7.1/testcontainers-bom-1.7.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.7.2/testcontainers-bom-1.7.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.7.3/testcontainers-bom-1.7.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.8.0/testcontainers-bom-1.8.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.8.1/testcontainers-bom-1.8.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.8.2/testcontainers-bom-1.8.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.8.3/testcontainers-bom-1.8.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.9.0/testcontainers-bom-1.9.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.9.1/testcontainers-bom-1.9.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/index.html (GET,404)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/maven-metadata.xml (GET,200)": 1,
...
[2024-03-07T08:55:33.850Z]        },
[2024-03-07T08:55:33.850Z]        "hostStats": {
[2024-03-07T08:55:33.850Z]          "api.github.com": {"requestCount": 14, "requestAvgMs": 227, "queueAvgMs": 2},
[2024-03-07T08:55:33.850Z]          "auth.docker.io": {"requestCount": 4, "requestAvgMs": 886, "queueAvgMs": 4},
[2024-03-07T08:55:33.850Z]          "ghcr.io": {"requestCount": 7, "requestAvgMs": 219, "queueAvgMs": 26},
[2024-03-07T08:55:33.850Z]          "gitlab.xdev.cloud": {
[2024-03-07T08:55:33.850Z]            "requestCount": 7,
[2024-03-07T08:55:33.850Z]            "requestAvgMs": 757,
[2024-03-07T08:55:33.850Z]            "queueAvgMs": 4
[2024-03-07T08:55:33.850Z]          },
[2024-03-07T08:55:33.850Z]          "index.docker.io": {"requestCount": 8, "requestAvgMs": 551, "queueAvgMs": 8},
[2024-03-07T08:55:33.851Z]          "repo.maven.apache.org": {
[2024-03-07T08:55:33.851Z]            "requestCount": 3092,
[2024-03-07T08:55:33.851Z]            "requestAvgMs": 21,
[2024-03-07T08:55:33.851Z]            "queueAvgMs": 90
[2024-03-07T08:55:33.851Z]          }
[2024-03-07T08:55:33.851Z]        },

[AB-xdev]

Ok so I did a bit of further research and found out the following:

1.

There is an environment variable called RENOVATE_EXPERIMENTAL_NO_MAVEN_POM_CHECK that can be used to disable the HEAD request checks.

I enabled it, now my Renvoate needs 30s for dependency resolution instead of 90s and sends 100 requests to Maven Central instead of ~3k.

2.

I think that you should get completely rid of the HEAD request check:
Historically the HEAD request was introduced with #5614 (as "does artifact exist"-check).

This PR addresses #4471 and renovatebot/config-help#529 which are both cases where people are using a misconfigured Maven Repository where the Artifact Level maven-metadata.xml files are corrupted (likely someone deleted the artifacts manually and forgot to update the metadata file).

IMHO the solution here is to fix the broken maven-metadata.xml on the server and it's not Renovates job to workaround it by sending a ton of HEAD requests.

If anybody tries to tell's you "no it's not broken" tell them that they should go read the Maven Specs as it clearly describes:

The artifactId level metadata (found in directory of artifactId) serves purpose of version discovery. This metadata contains list of versions of given GA coordinates.

[rarkins]

Let's switch it to opt in for v38. Can you create the PR?

[zharinov]

However, before we turn it off, we could do great job of detecting all the broken metadata files we've encountered across Maven central

[rarkins]

Maybe we can build another crowdsourced list built into the repo so users can add to it

[zharinov]

I'll take a look how could we pre-populate it based on our current warm cache...

[rarkins]

I think it would be useful to print information about how many versions were found to be invalid, and then if we wish to build from that data then we could log them too so that we can pull out and deduplicate the list from the Mend backend to bootstrap this list.

[AB-xdev]

Update Nov 2024:

• The situation has slightly improved with f182fb3
• The general problems still persist. I created feat(datasource/maven): Introduce option to assume that looked up maven data is specification compliant #32651 which should address them.