Garbage Collector on the ci-jenkins-io-artifacts S3 storage account

[lemeurherve]

Service(s)

AWS, ci.jenkins.io

Summary

Follow-up of #3643, we need to put in place a garbage collector to delete old artifacts and stashes from this S3 storage account used by ci.jenkins.io agents.

Note: the associated costs are low, less than 1$/month, this issue isn't urgent.

Reproduction steps

No response

[dduportal]

As per #3954, this issue is not needed anymore as the S3 bucket was deleted.

[jglick]

So where are artifacts & stashes being sent now; and is there a cleanup policy on that bucket?

[dduportal]

So where are artifacts & stashes being sent now; and is there a cleanup policy on that bucket?

Back to plain old data disk (eg. In the jenkins home of ci.jenkins.io). As we moved all agents to azure and the controller is also in azure, performances are way better!

as such, cleanup policy is tied to build retention policy

[jglick]

In the jenkins home

😱

[dduportal]

In the jenkins home

😱

It represents around 160gb (due to build cleanups), it is not that much.
Besides, jenkins home is on a local nvme at 25000 IOPS so no risks even for ci.j (spoiler : we don’t pay for it until end of august)

[jglick]

That is not the only consideration; if you do not use an artifact manager, stashes and artifacts must be streamed over the Remoting channel, putting CPU and I/O load on the controller.

[dduportal]

That is not the only consideration; if you do not use an artifact manager, stashes and artifacts must be streamed over the Remoting channel, putting CPU and I/O load on the controller.

Absolutely. the VM hosting ci.jenkins.io was sized for this case (default with artifacts on the JENKINS_HOME) and we never shrinked it after switching from default to S3 artifact manager.
To be transparent, the VM is huge and oversized for the usage: 8 vCPUs, 32 Gb memory.

We've carefully followed up the metrics after the switch, particularly with the huge amount of BOM builds in the days following the changes (18-19 May).

Result is that we don't see any additional load, eventually by peak (but even checking a 15 min period during a BOM build with a lot of stash/unstash does not show additional load). I believe the usage done by these builds and the oversize of the VM covers it.

Of course, we can afford this for ci.jenkins.io because the VM is running under credits until end of year (Azure credits until August, then AWS credits until December). We'll have to consider this again once it will be migrated into a paid account of course: but it is an optimization with lower priority for now: we'll add the S3 artifact manager back when we'll move it back to AWS.