roachtest: codify longer ttl external storage buckets

[msbutler]

Previously, the only codified external buckets for roachtests to back up to was
the cockroachdb-backup-testing buckets in s3 and gcs which each had a ttl of
1 day. This low ttl is not suitable for roachtests that produce backups that
the test failure investigator may want to inspect. This patch codifies the new
cockroachdb-backup-testing-long-ttl buckets in s3 and gcs, which currently
have a ttl of 20 days, the same ttl that team city artifacts have.

This patch also points the c2c, backup-restore/mixed-version, and
disagg-rebalance roachtests to use these new buckets.

Note this PR only points roachtests that run in public TC environments to the
new buckets. A future PR will set the BACKUP_TESTING_BUCKET_LONG_TTL env var
for private roacttests to a new bucket with a longer ttl.

Epic: none

Release note: none

[cockroach-teamcity]

This change is 

[msbutler]

unrelated extended ci flake with the bench job

[renatolabs]

This is great, thanks for working on this!

A couple of small requests:

• Can we update build/teamcity/internal/cockroach/nightlies/private_roachtest.sh to set the new env var there as well? We should have a similar setup in that project, and tests that run on the private config should not write to the public buckets.
• Can we update the mixed-version backup-restore test to use the long TTL bucket?

Also, I don't see a TTL configured for the "long TTL" version of the bucket on GCS or for the "short TTL" version of the bucket on AWS. Shouldn't we do it before we merge this?

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @DarrylWong, @itsbilal, and @msbutler)

---

-- commits line 9 at r1:
FWIW, TC is configured to keep roachtest artifacts for 20 days. Not that I'm suggesting we mirror it in the bucket necessarily, but maybe we amend the commit message to avoid confusion.

[renatolabs]

Actually the GCS "long TTL" bucket has a TTL of 30 days which is not aligned with what the code is saying.

[itsbilal]

Aside from Renato's comments, this LGTM from a storage perspective.

[msbutler]

Thanks for the review! Some notes:

• I've changed the ttl for these buckets to 20 days, to align with our tc retention policy. I've updated the commit message and comments to reflect this.
• I added that env var to privated_roachtest.sh. Do i need I populate that env var somewhere in TC as well?
• mixed version backup restore now uses the long ttl bucket as well
• I was planning to delete and remake the short ttl aws bucket a couple days after this merges since it lives in us-east-1 and it should live in us-east-2. I'm hoping we don't need to do a live migration 🙏

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @DarrylWong and @renatolabs)

---

-- commits line 9 at r1:

Previously, renatolabs (Renato Costa) wrote…

FWIW, TC is configured to keep roachtest artifacts for 20 days. Not that I'm suggesting we mirror it in the bucket necessarily, but maybe we amend the commit message to avoid confusion.

huh. It seems most intuitive to mimic how long tc artifacts are around so i changed the ttl's to be 20 days.

[renatolabs]

No changes needed in TC, but the private_roachtest script should set a long TTL bucket as well:

cockroach/build/teamcity/internal/cockroach/nightlies/private_roachtest.sh

Line 13 in 82ea947

export BACKUP_TESTING_BUCKET="cockroach-backup-testing-private"

I see that the "short TTL" bucket above has no TTL. Do you mind setting a 1-day TTL on that bucket, creating a new, 20-day TTL bucket on the private project (e2e-infra-381422), and setting BACKUP_TESTING_BUCKET_LONG_TTL in that script as well?

[msbutler]

I couldn't update/create buckets in the private gce project, so I left a todo and updated the commit message to reflects this. Opening up an internal jira ticket for dev inf.

[msbutler]

TFTR!

bors r=renatolabs

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[renatolabs]

@msbutler should we backport this to 23.1?

[msbutler]

thanks for the reminder! over here #111086