ROX-26276: update DNS after cluster reassignment

[johannes94]

Description

• Change FM to send UPSERT type requests to Route53, this way we can Update and Create CNAME records with the same operation
• Change Admin Endpoint for assign-cluster to reset route creation DB fields
• Change Admin Endpoint for assign-cluster to reset tenant to "provisioning" which will trigger resending of routes from FS -> Update in DB -> Upsert operation through cname_mgr -> Updated CNAME record after migration
• Added an additional DB field EnteredProvisioning to work around timeout issues, since CreatedAt is not a valid reference time for provisioning after reassignment. Adjusted preparing and provisioning operations to set and act according to the new field if present.
• Depends on ROX-26276: replace dinosaurs with centrals for cname components #2056 and ROX-26275: cluster reassignment admin API #2053

Checklist (Definition of Done)

• Unit and integration tests added
• Added test description under Test manual
• Documentation added if necessary (i.e. changes to dev setup, test execution, ...)
• CI and all relevant tests are passing
• Add the ticket number to the PR title if available, i.e. ROX-12345: ...
• Discussed security and business related topics privately. Will move any security and business related topics that arise to private communication channel.
• Add secret to app-interface Vault or Secrets Manager if necessary
• RDS changes were e2e tested manually
• Check AWS limits are reasonable for changes provisioning new resources
• (If applicable) Changes to the dp-terraform Helm values have been reflected in the addon on integration environment

Test manual

• Adjusted integration tests to test DNS reset on AssignCentralCluster Admin API as well
• Tested on an infractl OSD as well:
  • Setup infra cluster
  • make deploy/bootstrap deploy/dev
  • Set RHACS_CLUSTER_MIGRATION=true on FM
  • Set --enable-central-external-certificate=true on FM
  • Create a central
  • make sure DNS records are properly created
  • connect to the FM database to inspect routes_creation_id
  • Call the AssignCluster admin API endpoint (reassign to current cluster)
  • See in DB that all routes field are reseted and tenant is back to "provisioning" state
  • Verify tenant reaches ready state again, view that a new ID is stored in "routes_creation_id"
  • Verify that the change in AWS route53 has the correct timestamp (when new routes should be created)
  • Make sure to delete the tenant so that AWS resources are cleaned up

[ebensh]

Change Admin Endpoint for assign-cluster to reset tenant to "provisioning" which will trigger resending of routes from FS -> Update in DB -> Upsert operation through cname_mgr -> Updated CNAME record after migration

This is the one that makes me nervous. It's adding complexity to the state flow that other pieces may not be expecting - i.e. in the future, whenever we're reasoning about Provisioning and Ready, it's no longer a one-way transition.

[johannes94]

Change Admin Endpoint for assign-cluster to reset tenant to "provisioning" which will trigger resending of routes from FS -> Update in DB -> Upsert operation through cname_mgr -> Updated CNAME record after migration

This is the one that makes me nervous. It's adding complexity to the state flow that other pieces may not be expecting - i.e. in the future, whenever we're reasoning about Provisioning and Ready, it's no longer a one-way transition.

I didn't like that part as well. The alternative I thought of was to have FS always report the routes as opposed to only reporting routes if the tenant is not in ready state yet.

That would essentially make the state flow one-way but bloat every state communications request from FS to FM with additional data that is not necessary in most cases (if routes already created).

A third idea would be to adjust the FM <- FS polling API so that we additional to the state report the value of RoutesCreated boolean for a tenant and act based on that.

What approach would you choose or do you have a different idea?

[johannes94]

/retest

[johannes94]

/retest
cluster create failed

[johannes94]

/retest
cluster create failure

[ebensh]

LGTM, thanks for the changes!

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ebensh, johannes94

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ebensh,johannes94]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[johannes94]

/retest

[johannes94]

/retest