drtprod: emit audit logs for drt-chaos #126601
Conversation
This patch updates the Fluent Bit configuration for DRT clusters to emit audit logs. This is only configured for the `drt-chaos` cluster for now, but can easily be expanded to other clusters in the future. This patch also updates the way Fluent Bit sets the hostname on logs since Fluent Bit has not cut a release that contains the fix for fluent/fluent-bit#8971. Finally, this patch also fixes an issue with the Datadog agent configuration removal throwing a shell error. Epic: CLOUDOPS-9609 Release note: None
sudomateo
requested review from
DarrylWong and
renatolabs
and removed request for
a team
|
This change is |
| @@ -71,6 +70,7 @@ func Install(ctx context.Context, l *logger.Logger, c *install.SyncedCluster, co | |||
| DatadogSite: config.DatadogSite, | |||
| DatadogAPIKey: config.DatadogAPIKey, | |||
| DatadogService: config.DatadogService, | |||
| Hostname: vm.Name(c.Name, int(node)), | |||
There was a problem hiding this comment.
What will these hostnames look like? Hopefully globally unique?
There was a problem hiding this comment.
The format is CLUSTER-HOST. For example drt-large-0001, drt-large-0002, etc. Technically not globally unique, but roachprod doesn't allow you to create multiple clusters of the same name.
| apikey: ${dd_api_key} | ||
| dd_source: audit | ||
| dd_service: drt-cockroachdb | ||
| dd_tags: env:development,cluster:${cluster%:*},service:drt-cockroachdb,team:drt |
There was a problem hiding this comment.
Remind me what ${...%:*} means in bash?
There was a problem hiding this comment.
It removes everything from : to the end of the string. For example, given cluster=drt-large:1 then ${cluster%:*} will evaluate to drt-large. Operators can pass entire clusters (e.g., drt-large) or specific hosts (e.g., drt-large:1) to roachprod commands. This just gives us a clean, but admittedly obscure, way to retrieve the cluster name regardless of how the commands are run.
|
bors r+ |
|
This PR was included in a batch that was canceled, it will be automatically retried |
This patch updates the Fluent Bit configuration for DRT clusters to emit audit logs. This is only configured for the
drt-chaoscluster for now, but can easily be expanded to other clusters in the future.This patch also updates the way Fluent Bit sets the hostname on logs since Fluent Bit has not cut a release that contains the fix for fluent/fluent-bit#8971.
Finally, this patch also fixes an issue with the Datadog agent configuration removal throwing a shell error.
Epic: CLOUDOPS-9609
Release note: None