Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: grafana annotations read creds from cloud storage #125571

Merged
merged 1 commit into from
Jun 24, 2024

Conversation

DarrylWong
Copy link
Contributor

@DarrylWong DarrylWong commented Jun 12, 2024

As of #124099 we now store the service account creds in cloud storage. We already use this to access prometheus when generating dynamic configs. This change does the same for Grafana annotations by extracting the common logic into a helper.

This will allow users to have access to Grafana annotations out of the box locally, and limit the amount of benign but potentially confusing warnings about invalid credentials.

Release note: none
Fixes: none
Epic: none

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@DarrylWong DarrylWong force-pushed the annotation-secret-manager branch 2 times, most recently from ba372dc to a976db7 Compare June 12, 2024 19:50
@DarrylWong DarrylWong requested a review from nameisbhaskar June 12, 2024 19:58
@DarrylWong DarrylWong marked this pull request as ready for review June 12, 2024 19:59
@DarrylWong DarrylWong requested a review from a team as a code owner June 12, 2024 19:59
@DarrylWong DarrylWong requested review from herkolategan and removed request for a team June 12, 2024 19:59
@DarrylWong
Copy link
Contributor Author

@nameisbhaskar I cleaned up some of the logging and changed the error handling a bit in SetPromHelperCredsEnv. Basically it didn't make sense to me to continue in the case of missing credentials since the oauth token retrieval will fail that case? But lmk if that seems sound to you.

@DarrylWong DarrylWong force-pushed the annotation-secret-manager branch from a976db7 to 5b55193 Compare June 12, 2024 20:05
@DarrylWong DarrylWong changed the title roachtest: grafana annotations read creds from secret manager roachtest: grafana annotations read creds from cloud storage Jun 12, 2024
@DarrylWong DarrylWong force-pushed the annotation-secret-manager branch from 5b55193 to b32947f Compare June 13, 2024 15:35
Copy link
Contributor

@renatolabs renatolabs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM in terms of code reuse; just have some naming questions.

This will allow users to have access to Grafana annotations out of the box locally

I'm missing something in my understanding of how things are set up -- why does this change imply that everyone will have access out of the box? Doesn't it still require users to set environment variables locally?

grafanaAudience := os.Getenv(ServiceAccountAudience)
if grafanaAudience == "" {
return nil, errors.Newf("%s env variable was not found", ServiceAccountAudience)
if _, err := promhelperclient.SetPromHelperCredsEnv(ctx, false); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation for newGrafanaClient could be updated to mention the function that reads credentials from env (maybe SetPromHelperCredsEnv), as ServiceAccountJson no longer exists.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, the code reuse is good, but the fact that Grafana authentication works by calling promhelperclient.SetPromHelperCredsEnv is confusing because it suggests promhelperclient is involved.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes good point, maybe this should live in a separate identity package since this is really non specific to prom/grafana/etc., just anything that uses google idp.

serviceAccountJson = "PROM_HELPER_SERVICE_ACCOUNT_JSON"
serviceAccountAudience = "PROM_HELPER_SERVICE_ACCOUNT_AUDIENCE"
ServiceAccountJson = "PROM_HELPER_SERVICE_ACCOUNT_JSON"
ServiceAccountAudience = "PROM_HELPER_SERVICE_ACCOUNT_AUDIENCE"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these have more generic names? It's strange to me that we'll need to say that in order to use roachprod grafana-annotation, you'll need to set the PROM_HELPER_SERVICE_ACCOUNT_JSON env var.

@DarrylWong
Copy link
Contributor Author

Doesn't it still require users to set environment variables locally?

SetPromHelperCredsEnv does the following in order until one succeeds:

  1. Read in the env vars
  2. Check if promCredFile exists
  3. Read from cloud storage

In the case of number 3, it will set the environment variables locally after reading. It's a bit weird to me that the function doesn't just return the credentials and we need to refetch from the env, but I kept it as is to minimize changes.

@DarrylWong DarrylWong force-pushed the annotation-secret-manager branch 3 times, most recently from 9a8dc1f to 788b1bc Compare June 21, 2024 21:49
@DarrylWong
Copy link
Contributor Author

I've extracted the common logic into a roachprodutils package and renamed them all to be generic to google idp so hopefully that helps remove the confusion.

@DarrylWong DarrylWong force-pushed the annotation-secret-manager branch from 788b1bc to 3953c8f Compare June 21, 2024 22:30
As of cockroachdb#124099 we now store the service account creds in cloud
storage. We already use this to access prometheus when generating
dynamic configs. This change does the same for Grafana annotations
by extracting the common logic into a helper.

This will allow users to have access to Grafana annotations out
of the box locally, and limit the amount of benign but potentially
confusing warnings about invalid credentials.

Epic: none
Fixes: none
Release note: none
@DarrylWong DarrylWong force-pushed the annotation-secret-manager branch from 3953c8f to 079a0f5 Compare June 24, 2024 14:05
@DarrylWong
Copy link
Contributor Author

TFTRs!

bors r=herkolategan, renatolabs

@craig craig bot merged commit 6693f4a into cockroachdb:master Jun 24, 2024
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants