Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to set the 'InfraID' with an environment variable. #5101

Conversation

mmartinv
Copy link

@mmartinv mmartinv commented Jul 22, 2021

This change allows the user to set the OPENSHIFT_INSTALL_INFRA_ID_OVERRIDE environment
variable to modify the InfraID.

The contents of the OPENSHIFT_INSTALL_INFRA_ID_OVERRIDE correspond to the random part of
the InfraID so it must be formed by 5 random lowercase alphanumeric chars with no vowels,
otherwise an error will be returned.

An example on how to use it:

OPENSHIFT_INSTALL_INFRA_ID_OVERRIDE="xxxxx" openshift-install create manifests --dir=.

This change is needed to implement external provider support in the assisted-installer project.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 22, 2021

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign crawford after the PR has been reviewed.
You can assign the PR to them by writing /assign @crawford in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 22, 2021

Hi @mmartinv. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 22, 2021
@patrickdillon
Copy link
Contributor

This change is needed to implement external provider support in the
assisted-installer project.

Can you elaborate? It seems like this a dependency on something that is designed to be non-deterministic.

@@ -74,6 +75,15 @@ func generateInfraID(base string, maxLen int) string {
}
base = strings.TrimRight(base, "-")

// Return the InfraID provided by the "INFRA_ID" environment variable if valid
if envInfraID, ok := os.LookupEnv("INFRA_ID"); ok && envInfraID != "" {
validInfraIDString := fmt.Sprintf("^%s-([b-df-hj-np-tv-z0-9]){%d}$", base, randomLen)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe move this pattern to a constant and explain what are you looking for

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to explain more in the comment, moving the pattern to a constant make the code less readable IMO.

@Gal-Zaidman
Copy link
Contributor

This change is needed to implement external provider support in the
assisted-installer project.

Can you elaborate? It seems like this a dependency on something that is designed to be non-deterministic.

Sure in the installer the InfraID is used to set the names of the VMs and for oVirt it is also used to set a cluster tag (in oVirt).
We are working on an effort to introduce support for external providers (oVirt, vsphere, and anyone else) to the assisted installer.
In the assisted installer the user provisions the OCP nodes, that are discovered by the assisted installer service, and then the assisted installer service uses the openshift-installer to create the manifests and install config.
One of the earliest problems we encountered is that due to the random InfraID the names of the Nodes(in the manifest) are different from the nodes the user provisions, and since it is random we can't tell the user to create the Nodes with specific names.
Also, the InfraID is later used by the platforms in various ways to identify the cluster, for example in oVirt we use it to set an oVirt tag which helps us group the cluster resources in oVirt, and we rely on it in various places such as cluster destroy or CSI/Machine provider logic to filter cluster resources quickly and prevent unnecessary API calls.

Having a hook in the installer to set the InfraID prior to the installation will help us overcome does issues.

@mmartinv mmartinv force-pushed the allow-to-set-the-infra-id-with-an-env-variable branch from 07f4b28 to 62f4e42 Compare July 23, 2021 09:17
@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 23, 2021
@patrickdillon
Copy link
Contributor

/hold
I would like to hold this for some more discussion.

Has this been discussed in an enhancement/arch call? I am concerned about the dependencies we are introducing here. These topics have been discussed with regard to the cluster ID eg. #761 & #783 but I am not certain how this relates to the infra id.

@Gal-Zaidman
Copy link
Contributor

Gal-Zaidman commented Jul 25, 2021

/hold
I would like to hold this for some more discussion.

Has this been discussed in an enhancement/arch call?

Added it to my enhancement on assisted installer: https://github.com/openshift/assisted-service/pull/2272/files#diff-988c4e5d655472b2174837171f5d89978ca0f86ec8554329b386dccb9990ea83R134-R147

I am concerned about the dependencies we are introducing here. These topics have been discussed with regard to the cluster ID eg. #761 & #783 but I am not certain how this relates to the infra id.

Those two seem unrelated to me (unless I missed something) anyway I don't think we are adding dependencies, we keep the situation as is, but add a hook to preset it (while making sure that the structure is identical to the current state)

@patrickdillon
Copy link
Contributor

/hold
I would like to hold this for some more discussion.
Has this been discussed in an enhancement/arch call?

Added it to my enhancement on assisted installer: https://github.com/openshift/assisted-service/pull/2272/files#diff-988c4e5d655472b2174837171f5d89978ca0f86ec8554329b386dccb9990ea83R134-R147

I am concerned about the dependencies we are introducing here. These topics have been discussed with regard to the cluster ID eg. #761 & #783 but I am not certain how this relates to the infra id.

Those two seem unrelated to me (unless I missed something) anyway I don't think we are adding dependencies, we keep the situation as is, but add a hook to preset it (while making sure that the structure is identical to the current state)

Thanks I left a comment in the enhancement. The dependency is that we are creating reliable behavior through deterministic naming conventions so consumers may start to depend on resources with certain names. Although this is an environment variable it is essentially creating an API.

@Gal-Zaidman
Copy link
Contributor

/hold
I would like to hold this for some more discussion.
Has this been discussed in an enhancement/arch call?

Added it to my enhancement on assisted installer: https://github.com/openshift/assisted-service/pull/2272/files#diff-988c4e5d655472b2174837171f5d89978ca0f86ec8554329b386dccb9990ea83R134-R147

I am concerned about the dependencies we are introducing here. These topics have been discussed with regard to the cluster ID eg. #761 & #783 but I am not certain how this relates to the infra id.

Those two seem unrelated to me (unless I missed something) anyway I don't think we are adding dependencies, we keep the situation as is, but add a hook to preset it (while making sure that the structure is identical to the current state)

Thanks I left a comment in the enhancement. The dependency is that we are creating reliable behavior through deterministic naming conventions so consumers may start to depend on resources with certain names. Although this is an environment variable it is essentially creating an API.

Thanks now I understand your concerns a bit better, but I think that as long as the env var(or API) follows the same deterministic naming conventions then it shouldn't be a problem.
Yes then can be a situation in which a user will deploy 2 clusters with the same env var resulting in 2 clusters with the same infraID, but since it will not be documented for customers, includes a warning in the installation log, and is for our internal usage I think it is OK.
Worse and much harder to detect problems can occure with using the other 2 env vars ;)

@mmartinv mmartinv force-pushed the allow-to-set-the-infra-id-with-an-env-variable branch 2 times, most recently from d7015da to ec8b292 Compare July 26, 2021 08:43
This change allows the user to set the "OPENSHIFT_INSTALL_INFRA_ID_OVERRIDE"
environment variable to modify the "InfraID".

The contents of the "OPENSHIFT_INSTALL_INFRA_ID_OVERRIDE" correspond to
the random part of the InfraID so it must be formed by 5 random lowercase
alphanumeric chars with no vowels, otherwise an error will be returned.

An usage example would be:

 OPENSHIFT_INSTALL_INFRA_ID_OVERRIDE="xxxxx" openshift-install create manifests --dir=.

This change is needed to implement external provider support in the
assisted-installer project.
@mmartinv mmartinv force-pushed the allow-to-set-the-infra-id-with-an-env-variable branch from ec8b292 to 5ecddff Compare July 26, 2021 08:53
@staebler
Copy link
Contributor

staebler commented Oct 8, 2021

In general, the use of environment variables to drive configuration of the cluster is discouraged in the installer. The install-config.yaml is the place where configuration should live. We have allowed the use of environment variables in a few rare cases, where the alternatives have been onerous or where the environment variable is only for testing and not supported for use by the end user.

I am not yet convinced that this is one of the cases where an exception is justified. If we want to provide the option for the user to be able to use a pre-determine Infra ID, then we should add that to the install-config.yaml. However, It seems reasonable enough to me to have a user of the installer that needs the infra ID get it from the infrastructure manifest rather than taking the approach of the user supplying the infra ID.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Apr 1, 2022

@mmartinv: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-workers-rhel8 5ecddff link /test e2e-aws-workers-rhel8
ci/prow/e2e-alibaba 5ecddff link true /test e2e-alibaba
ci/prow/e2e-aws 5ecddff link true /test e2e-aws
ci/prow/e2e-gcp-upgrade 5ecddff link true /test e2e-gcp-upgrade
ci/prow/e2e-aws-upgrade 5ecddff link true /test e2e-aws-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@barbacbd
Copy link
Contributor

/close

@openshift-ci openshift-ci bot closed this May 26, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 26, 2022

@barbacbd: Closed this PR.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants