Past Release sites are taking long time to load

[yashcho]

Service(s)

get.jenkins.io

Summary

Below two links from page (https://www.jenkins.io/download/) are taking more than 30 seconds to load and show the table with list of old versions. Can you please look into it?

https://get.jenkins.io/war/

https://get.jenkins.io/war-stable/

Reproduction steps

1. Open browser by removing cache and cookies then hit any one of link.
2. Either of them will take more than 30 second or crash with status as "502 Bad Gateway"

[dduportal]

Hi @yashcho thanks for raising this issue, I can reproduce the issue, looking into it.

[dduportal]

First investigation:

• The Apache pod serving the files such as the HTML content on these pages (e.g. not when redirected to mirrors) restarted quite often since yesterday, while the redirector service (and reverse-proxy to the former) seems fine:

mirrorbits               mirrorbits-bb668f8f6-5bzqd                                        2/2     Running   0                 25h
mirrorbits               mirrorbits-bb668f8f6-hdxbs                                        2/2     Running   0                 25h
mirrorbits               mirrorbits-bb668f8f6-wlrxd                                        2/2     Running   0                 25h
mirrorbits               mirrorbits-files-76848c4d59-6cn5d                                 1/1     Running   371 (2m46s ago)   4d15h
mirrorbits               mirrorbits-files-76848c4d59-ds6bb                                 1/1     Running   192 (2m41s ago)   25h
mirrorbits               mirrorbits-files-76848c4d59-k959c                                 1/1     Running   194 (3m10s ago)   44h

• A quick check on the recent log lines with kubectl -n mirrorbits -l app.kubernetes.io/name=mirrorbits-files logs shows 2 kind of errors:

  • Permission error on the (Azurefile) filesystem:

AH00132: file permissions deny server access: /usr/local/apache2/htdocs/war/HEADER.html

• Apache overload:

AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

[dduportal]

• Triggering a node pool image upgrade to force recylcing the underlying VMs (and the associated azurefile mount) as a short ter measure

[dduportal]

• The service was upgraded, with a 2 minut outage => no more performances issues (no more timeouts)
  • Ref. hotfix(prodpublick8s) migrate mirrorbits PV to native CSI, use a 1000 Gib size and fine-tune mountoptions kubernetes-management#3862
• The /war-stable link is now working very well
  • It lists LTS releases as expected
• There is still an issue with the /war link: it appears that HTML page is missing some partials, work in progress
  • Direct WA download still works as expected

[dduportal]

Operations notes for the infra-team:

Preliminaries:

• A snapshot of the persistent volume was taken before operations (named 20230419-09h46-utc)
• Checked that all involved PVs are in "retain" mode in the mirrorbits namespace
• Removed all the PVCs and PVs related to redis as mirrorbits uses an Azure-managed Redis instances since 2.5 years
  • The associated Azure disks are not deleted and should be once prodpublick8s will be deleted, after migration to publick8s

Operations:

• Removed namespace "mirrorbits" to ensure that the PVC mirrorbits-binary is removed properly
• Removed the PV mirrorbits-binary with kubectl delete pv mirrorbits-binary as it's immutable
• Applied manually the Helm chart to re-create the PV, PVC and pod

Checks:

• Service was back after 2 minutes of outage
• Files are mounted, and mirror redirections are present
• The reported file volume size is now 1000 Gib

[dduportal]

• The performances issues are back when serving the content of /war and /war-stable
• The access logs shows that we are hammered by GET requests to these to paths

[dduportal]

• The HEADER.html and FOOTER.html files are used by an Apache feature named "FancyIndex": Apache renders the HTML page with these 2 partial templats, and insert in the middle a dynamic directory listing. Ref:

  • IndexGenerator python script now generate html for Apache fancy directory jenkinsci/packaging#159
  • Add site to pkg.jenkins.io staging jenkinsci/packaging#171

• Checking on the reference VM pkg.origin.jenkins.io:

  • Retrieved locally the 2 files and copying both in the Azure Blob Storage (for the /war subdirectory only)
  • Also, the synchronisation log written by the sync.sh shows an error: fix published in the commit jenkins-infra/mirror-scripts@bebd767 (fixing up fix(sync recent releases): adding a venv python for blobxfer mirror-scripts#10 and fix(batch upload,sync) follow up of 503b7ac to use a virtual env for blobxfer CLI mirror-scripts#11)
  • Triggered the sync.sh script as user mirrorbrain (as per its crontab)

Let's see the result

[dduportal]

• Synchonisation successful
• The Apache errors disappeared with fix(prodpublick8s) tune mirrorbit to fit resource usage kubernetes-management#3863

The performances seems ok for the /war-stable/ page but not for the /war.

[yashcho]

@dduportal, Even for me it is the case what do you suggest?

[dduportal]

@dduportal, Even for me it is the case what do you suggest?

I'm not sure to understand, what is the exact problem you are having?

It is true that he page https://get.jenkins.io/war/ is slow, but we do not see HTTP/5XX errors on it.

Do you still see errors are are you blocked by the "slow" response time of this page? If you are block, can you describe what you are trying to achieve (as we could provide alternative and way more efficient solutions).

We are working on this matter but it is not the top priority as nothing seems blocked or broken: happy to reevaluate with more details

[yashcho]

are you blocked by the "slow" response time of this page?

Yes, basically I am trying to access the URL and fetch the list of versions from table programmatically. So the test cases are failing due to slow response to fetch the table details.

[dduportal]

are you blocked by the "slow" response time of this page?

Yes, basically I am trying to access the URL and fetch the list of versions from table programmatically. So the test cases are failing due to slow response to fetch the table details.

In that case, you might be interested by checking the artifactory metadatas. We have an example here, in shell script, which checks for the 2 past LTS releases: https://github.com/jenkinsci/docker/blob/master/.ci/publish.sh#L85-L89 and the 2 past weekly releases: https://github.com/jenkinsci/docker/blob/ca53c743e9e83db0cd235723e6689cd1490e1d13/.ci/publish.sh#L79-L83

The URL https://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/maven-metadata.xml should be considered as the source of truth AND it's XML so it's made for programamtic access (while an HTML dynamically generated page is made for humans)

[yashcho]

Do you have similar pattern using golang codebase instead of shell script?

[dduportal]

Do you have similar pattern using golang codebase instead of shell script?

No, that way out of our scope. Do not hesitate to propose a contribution if you find or build something.

[yashcho]

Sure..

[dduportal]

Hello @yashcho , do you still have the problem or can we close this issue?

[timja]

This seems far quicker now since #3917

Plugins list is still slow-ish but does load eventually:
https://get.jenkins.io/plugins/

[lemeurherve]

Plugins list is still slow-ish but does load eventually: https://get.jenkins.io/plugins/

Off-topic & low priority: should an index be generated like jenkinsci/packaging#159 to avoid Apache having to render this particular folder list?

[dduportal]

This seems far quicker now since #3917

Plugins list is still slow-ish but does load eventually: https://get.jenkins.io/plugins/

Yes, far quicker, but still take a bit of time. At least we are not reaching timeout.

Wasn't able to switch to NFS PVC (failed to some timeout in CSI driver when trying to reach server: need to be dig up) though.

Off-topic & low priority: should an index be generated like jenkinsci/packaging#159 to avoid Apache having to render this particular folder list?

I believe it is totally on the topic here as it could properly solve the problem here: generated files would be mirrored and spread across the world. And even if served by the HTTPD fallback (or achives), still better than today!

cc @MarkEWaite as he mentionned this possibility a few weeks earlier.