[DDO-1292] Rawls Liquibase Migration Pre-sync Job

[jack-r-warren]

Summary of DDO-1292:

• New liquibase.properties file produced by firecloud-develop (DDO-1297)
  • Rationale: Rawls's DB config is in rawls.conf produced by firecloud-develop, but we can't read that from CLI
  • PR: https://github.com/broadinstitute/firecloud-develop/pull/2565
  • Currently deployed to dev
  • In the future, DB config would be helm-templated from single source of truth
• K8s job (and related resources) are applied before rest of Rawls in ArgoCD sync, running Liquibase via the app's image before the app ever starts
  • Rationale: Now the app doesn't need to run Liquibase migrations on startup--the job runs to completion and kills its own CloudSQL proxy sidecar to let the ArgoCD sync continue (DDO-1301)
• Values file enabling this lives over in terra-helmfile, that's where we'd make it do more than a dry-run
  • PR: https://github.com/broadinstitute/terra-helmfile/pull/1349
  • After this merge that PR will be updated to refer to released helm chart
  • Currently deployed to dev

[mikeflinn]

Awesome work @jack-r-warren! looks great overall. Just a few small questions and nits.

[choover-broad]

This is a seriously fantastic first Helm PR @jack-r-warren! 💯🥇

(Left a few comments inline, it may be that I'm just missing context)

[choover-broad]

Do we need separate RBAC resources for this job? I think the permissions will be very similar to whatever the Rawls application uses. (Apologies if there's already been some background discussion here that I missed!)

Seems like we could add PreSync to the existing resources to make sure they exist when the job is run, but not delete them before every sync?

[jack-r-warren]

We don't strictly need separate RBAC, but we do need them to exist, and unfortunately: argoproj/argo-cd#3502 (comment)

Argo CD hooks are not considered part of Argo CD application resources, and features like drift detection and health assessments do not work. So the solution should involve not having to resort to adding "helm.sh/hook": pre-install to a Namespace resource.

Despite the clutter that this involves (#387 (comment)) I don't know of a better solution

[choover-broad]

Does this comment offer a viable workaround? argoproj/argo-cd#3502 (comment)

I.e.

• RBAC resources sync in wave -2
• Job syncs in wave -1 with a Sync hook
• Everything else syncs in default wave 0

If that won't work, I'm fine with duplicating the RBAC resources!

[jack-r-warren]

I hadn't seen that, that's a good idea. I'd talked to Mike and just now put the RBAC stuff behind a flag so the job can reuse the existing one. I think the only downside to leaning on sync waves is that the normal rbac resources would need to have that sort of migration-specific attribute, but that's a small downside imo. Let's chat at standup

[jack-r-warren]

This worked!

[choover-broad]

Nice, looks great to me!

The values allow us to enable this in dev only for an arbitrary length of time, right?

[jack-r-warren]

Nice, looks great to me!

The values allow us to enable this in dev only for an arbitrary length of time, right?

Correct! Within each environment the values can control whether it is enabled (default false), whether it is dry run (default true), and whether failures will fail the sync (default true)