Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VC-36032] Configure controller-runtime to log with klog #626

Merged
merged 1 commit into from
Nov 21, 2024

Conversation

wallrj
Copy link
Member

@wallrj wallrj commented Nov 20, 2024

I assume that the controller-runtime logs were being discarded.
With this change I hope to find some controller-runtime logs among the e2e test logs.

Testing

$ make test-e2e-gke
...
{
  "ts": 1732123466929.3813,
  "caller": "metrics/metrics.go:97",
  "msg": "cache not synced yet, skipping metrics venaficonnection_*",
  "source": "controller-runtime",
  "v": 0
}
...
{"ts":1732123472568.4834,"caller":"agent/run.go:409","msg":"Data sent successfully","v":0,"logger":"Run.gatherAndOutputData.postData"}

That single message seems to come from venafi-connection-lib, so perhaps it's misleading to label these logs with source: "controller-runtime":

@maelvls Does that log message indicate a bug in the way we're instantiating the venafi-connection-lib client?

// TODO(mael): The rest of the codebase uses the standard "log" package,
// venafi-connection-lib uses "go-logr/logr", and client-go uses "klog". We
// should standardize on one of them, probably "slog".
ctrlruntimelog.SetLogger(logr.Logger{})
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -10,3 +10,4 @@ authentication:

extraArgs:
- --logging-format=json
- --log-level=2
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed this so that could see trace level logs in the e2e test.
As it happens, the single controller-runtime log message from venafi-connection-lib is at level 0, so this change is not strictly necessary.

@maelvls
Copy link
Member

maelvls commented Nov 21, 2024

cache not synced yet, skipping metrics venaficonnection_*

I remember adding this log line in venafi-connection-lib's metrics.go (internal link). I remember writing this:

// We found a niche problem where `/metrics` would hang forever in case of a
// misconfigured RBAC. This was due to `cache.List` hanging until the cache
// is synced. To prevent that, we skip reporting this subset of the metrics.
if !hasSynced(cc.cache) {
	cc.log.Info("cache not synced yet, skipping metrics venaficonnection_*")
	return
}

I don't know why WaitForCache is taking so long to return. The Helm chart's RBAC is correct, so it shouldn't be this... This needs debugging, but we can do that in the next PR.

@wallrj wallrj merged commit eb3c30a into master Nov 21, 2024
2 checks passed
@wallrj wallrj deleted the VC-36032/controller-runtime-logging branch November 21, 2024 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants