Catalog and patch servers

[jdno]

The infra-team operates a handful of physical and virtual servers. These are on various different patch levels, with some running unsupported software versions. We should go through our infrastructure, catalog the machines that we operate and their operating systems, and update the installed software packages.

For machines that run unsupported operating systems, e.g. Ubuntu 18.04, we should plan and perform a manual upgrade to the next supported version.

The list of machines and their OS versions can be used in a following task to ensure all servers run on the same/latest operating system.

[MarcoIeni]

I started working on this.

Things to check

• add other things to check

Patch servers

• playground
  • provision new ec2
  • configure new ec2
  • use new ec2
  • delete old ec2 delete play-1 EC2 instance simpleinfra#591
• dev desktops
• monitoring https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/.E2.9C.94.20Monitoring.20server.20update.20strategy
• bastion
  • provision new ec2 feat: create ubuntu24 bastion ec2 simpleinfra#584
  • configure new ec2
  • use new ec2 swap bastion EC2 simpleinfra#598
  • use new ec2 for real (bastion2 was pointing to the first ec2) swap domain of bastion 1 and 2 simpleinfra#619
  • delete old ec2 chore: delete old bastion EC2 simpleinfra#623
  • remove old ip from allow list of other machines
• crater https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/Crater.20ubuntu.2024.20update
  • update dockerfile chore(docker): update to ubuntu 24 crater#741
  • update agents vm chore(crater): update ubuntu image of agents simpleinfra#599
  • update vm
  • fix ansible issue fix(crater): install awscli from snap simpleinfra#607
• monitorbot update dockerfile to ubuntu 24 monitorbot#20
• docs-rs: don't update because we have a migration plan

GCP

• check resources

Ansible

• check ansible

Azure

• check azure
• crater-azure-1 and crater-azure-2 chore: remove references to crater-azure simpleinfra#622

AWS

For the following accounts, cross-reference what's in aws and terraform (in all regions). Using tag explorer or cost explorer might help.

AWS accounts:

• bors-prod. docs: document bors #123. Used regions:

  • us-east-1
  • us-east-2

• crates-io-prod (region us-east-1). It's basically empty. docs: document crates-io-logs #129

• dev-desktops-prod. Already covered in the docs.

  • should I update the operating systems?

• docs-rs-staging. I'm not documenting this because it's unused.

• rust admin (aka legacy)

  • terragrunt:

    • crates-io-prod docs: document the rest of crates-io infra #130
    • datadog-aws docs: link datadog integrations #131
    • datadog-fastly docs: link datadog integrations #131
    • releases-prod docs: document releases #132
    • rustup-prod docs: document rustup #137

  • terraform:

    • bastion docs: document bastion #139
    • bors docs: document homu #138
    • crater. document crater #157
    • crates-io-heroku-metrics docs: document the rest of crates-io infra #130
    • dev-desktops already documented
    • discord-mods-bot docs: document discord mods bot #141
    • dns docs: document dns #153
    • dns-delegation docs: document dns #153
    • docs-rs docs: document docs.rs #158
    • domain-redirects document domain-redirects #156
    • fastly-exporter docs: document fastly-exporter #143
    • monitorbot docs: document monitorbot #149
    • playground docs: document rust playground #144
    • rds-databases added a readme in the terraform folder in docs: document the rds-databases module simpleinfra#577
    • releases docs: document releases #132
    • rust-forge docs: document rust-forge #146
    • rust-log-analyzer docs: document rust-log-analyzer #154
    • rustc-ci docs: document rust ci #161
    • rustc-perf docs: document rustc-perf #150
    • team-members-access docs: document team-members-access #148
    • team-members-datadog docs: document fastly and datadog team members #147
    • team-members-fastly docs: document fastly and datadog team members #147
    • team-repo docs: add sync-team to the service catalog #124

• sync-team-prod (us-east-1, us-east-2). I'm not documenting this because it's unused from what JD told me.

Personal notes

Tasks

Ansible configures the things you provision in terraform.

• look at simpleinfra/ansible/playbooks

To remember

I found useful to:

1. download csv from tag explorer
2. use nushell: open resources.csv | group-by Region

Questions

• why does bors-prod uses two regions?
  • we use one region for the terraform status. We want to keep the status in the same region for all the accounts.
  • East 2 is less utilized than East 1 because East 1 is the default.
• why aren't crates-io logs present in the crates-io-prod account?
  • Neverming, it should be there.
• Why do we have both https://github.com/rust-lang/simpleinfra/tree/master/terragrunt/accounts/sync-team-prod and https://github.com/rust-lang/simpleinfra/tree/master/terraform/team-repo ? Some files are almost the same, e.g. https://github.com/rust-lang/simpleinfra/blob/master/terraform/team-repo/sync-team.tf and https://github.com/rust-lang/simpleinfra/blob/master/terragrunt/modules/sync-team/sync-team.tf
• should we delete the dev-dekstop ec2 in the legacy account?
  • yes: chore: delete unused dev desktop simpleinfra#631
• is https://github.com/rust-lang/simpleinfra/blob/c22d3f0aabcaa8065238c6e233f1c92e348086c9/terraform/rustc-ci/environments.tf#L24-L25 used?
  • Maybe it's used by the security team. They have a policy to delete the data after a while, so that's probably why it's empty.