Add Alerts, Logging, Channels Factories #2758

joshw123 · 2024-12-10T17:13:50Z

Checklist

I applicable, I acknowledge that I have:

Read the contributing guide
Ran terraform fmt on all modified files
Regenerated the relevant README.md files using tools/tfdoc.py
Made sure all relevant tests pass

juliocc

@joshw123 this is great stuff!

I left comments below, but besides that:

We need to update project/README.md to include examples the new variables and factories
FAST tests need to be updated

I have a few more things, but let's start with this batch.

modules/project/variables.tf

modules/project/alerts-factory.tf

modules/project/README.md

modules/project/alerts-factory.tf

modules/project/logging-metrics-factory.tf

modules/project/variables-metrics-alerts.tf

Co-authored-by: Julio Castillo <[email protected]>

joshw123 · 2024-12-11T14:12:38Z

@joshw123 this is great stuff!

I left comments below, but besides that:

We need to update project/README.md to include examples the new variables and factories

FAST tests need to be updated

I have a few more things, but let's start with this batch.

Thank You, Ill look at the above, Im checking through all the fields too to ensure optional and required match the docs, and to include validation where necessary :)

joshw123 · 2024-12-13T13:47:02Z

@ludoo @juliocc The only two tests that are still not passing are Module Examples Test report

examples.test_plan.test_example[terraform:modules/project-factory:Example:1]
AssertionError: wrong number of resources
assert 56 == 59

examples.test_plan.test_example[terraform:modules/project-factory:Tests:1]
AssertionError: wrong number of resources
assert 22 == 25

I don't know if its just me, but I can't seem to see where it is getting them values, could you point me in the right direction please? :)

Thank You.

juliocc · 2024-12-13T14:19:36Z

examples.test_plan.test_example[terraform:modules/project-factory:Example:1] AssertionError: wrong number of resources assert 56 == 59

Look at the #tftest tag at the end of the code block in this section.

examples.test_plan.test_example[terraform:modules/project-factory:Tests:1] AssertionError: wrong number of resources assert 22 == 25

Same thing here.

Take a look at the contributing guide for more details.

ludoo · 2024-12-15T06:58:40Z

Can you sign the CLA from the ju****c@gmail.com account?

juliocc · 2024-12-15T10:12:40Z

Can you sign the CLA from the ju****c@gmail.com account?

That address is mine. Should be fixed now.

juliocc · 2024-12-16T14:29:27Z

@joshw123 I'll do my second pass today. Apologies for the delay.

ludoo · 2024-12-17T09:23:20Z

fast/stages/0-bootstrap/variables.tf

@@ -102,6 +102,12 @@ variable "custom_roles" {
  default     = {}
 }

+variable "default_alerts_email" {


should we use the existing essential_contacts email here?

I had a similar thought either using an existing group, or building something with var.organization.domain

I actually prefer using essential_contacts

ludoo · 2024-12-17T09:23:45Z

fast/stages/0-bootstrap/outputs.tf

@@ -23,7 +23,7 @@ locals {
        # If users give a list of custom audiences we set by default the first element.
        # If no audiences are given, we set https://iam.googleapis.com/{PROVIDER_NAME}
        audiences = try(
-          local.cicd_providers[v["identity_provider"]].audiences, ""
+          local.cicd_providers[v["identity_provider"]].audiences, "e"


ludoo · 2024-12-17T09:25:11Z

fast/stages/0-bootstrap/outputs.tf

@@ -109,6 +109,7 @@ locals {
        vpcsc       = module.automation-tf-vpcsc-sa.email
        vpcsc-r     = module.automation-tf-vpcsc-r-sa.email
      }
+      default_alerts_email = var.default_alerts_email


this should go in globals, but TBH I would not even pass this down as it's specific to stage 0

ludoo · 2024-12-17T09:25:31Z

fast/stages/0-bootstrap/outputs.tf

@@ -109,6 +109,7 @@ locals {
        vpcsc       = module.automation-tf-vpcsc-sa.email
        vpcsc-r     = module.automation-tf-vpcsc-r-sa.email
      }
+      default_alerts_email = var.default_alerts_email


should we export notification channels to the following stages?

ludoo · 2024-12-17T09:26:03Z

fast/stages/1-resman/variables-fast.tf

@@ -68,6 +68,12 @@ variable "custom_roles" {
  default = null
 }

+variable "default_alerts_email" {


this might be different for notifications set here, I would not include it in FAST variables

ludoo · 2024-12-17T09:26:35Z

fast/stages/2-network-security/variables-fast.tf

@@ -37,6 +37,12 @@ variable "billing_account" {
  }
 }

+variable "default_alerts_email" {


same as for resman, this should be configurable at the stage level

juliocc

Leaving a few more comments below.

I also sent you a PR against your branch with a few styling fixes. Once you get that merged I'll try to get started on removing the default_alerts_email variable and use essential_contacts

juliocc · 2024-12-12T16:30:31Z

fast/stages/0-bootstrap/data/logging-metrics/compliance.yaml

+
+network-firewall-config-changes:
+  description: "Monitor VPC network firewall configuration changes inside GCP projects"
+  filter: "resource.type=\"gce_firewall_rule\" AND (protoPayload.methodName:\"compute.firewalls.delete\" OR protoPayload.methodName:\"compute.firewalls.insert\")"


please use "|" syntax for all filters in this file

I've done this in the PR I sent

juliocc · 2024-12-12T16:31:01Z

fast/stages/0-bootstrap/outputs.tf

-  sensitive   = true
-  value       = local.tfvars_globals
+  sensitive   = false
+  value       = jsonencode(local.tfvars_globals)


What's this about?

/cc @ludoo

We mark large outputs as sensitive so as not to pollute the screen at every apply, as they still can be fetched explictly even when sensitive. So I would keep it.

As for jsonencode, we never do it for outputs and the globals file is JSON anyway, so I would also revert that.

juliocc · 2024-12-17T11:37:29Z

fast/stages/0-bootstrap/variables.tf

@@ -102,6 +102,12 @@ variable "custom_roles" {
  default     = {}
 }

+variable "default_alerts_email" {


I had a similar thought either using an existing group, or building something with var.organization.domain

I actually prefer using essential_contacts

juliocc · 2024-12-18T16:33:04Z

fast/stages/0-bootstrap/billing.tf

+    alerts          = var.factories_config.alerts
+    channels        = var.factories_config.channels
+    logging_metrics = var.factories_config.logging_metrics
+  }


now that we are creating LBMs and alerts, do we need to enable the logging and monitoring APIs?

Same, applies to all projects

juliocc · 2024-12-18T17:19:43Z

modules/project/alerts.tf

+  notification_channels = [
+    try(google_monitoring_notification_channel.this[each.value.notification_channels].id,
+    each.value.notification_channels),
+    try(google_monitoring_notification_channel.default[0].id, "")
+  ]


I'm not sure this works (you're indexing google_monitoring_notification_channel.this with each.value.notification_channels, which is a list). Please double check.

Also, I think the default shouldn't be used if there are notification channels defined by the alert. For me, the logic should be:

Try to use the locally created channel first,

If the specified channel id is not created by us, use it as as an id (i.e. the notification channel was created elsewhere)

If there are no notification channels, use the default, if provided.

joshw123 and others added 6 commits December 6, 2024 16:56

WIP: Logging Alerts Factory

6a8ae06

Implement Logging Alerts on Remaining Modules

2586c54

Merge branch 'master' into AlertsFactory

4e9f75e

Documentation & FMT

5b768a3

Convert To Multiple Factories

9ba92c3

Correct Project

4e8a745

github-actions bot added on:FAST on:modules labels Dec 10, 2024

joshw123 and others added 4 commits December 10, 2024 17:25

Update Documentation

8f16176

Merge branch 'master' into AlertsMetrics

44b0bcc

Merge branch 'master' into AlertsMetrics

82831ab

Merge branch 'master' into AlertsMetrics

f72d382

juliocc requested changes Dec 11, 2024

View reviewed changes

joshw123 and others added 2 commits December 11, 2024 13:34

Update modules/project/alerts-factory.tf

af1efa1

Co-authored-by: Julio Castillo <[email protected]>

Update fast/stages/0-bootstrap/data/logging-metrics/compliance.yaml

201912e

Co-authored-by: Julio Castillo <[email protected]>

joshw123 and others added 5 commits December 12, 2024 11:08

Merge branch 'master' into AlertsMetrics

8e89eb3

Update Tests, Resources

8c81fcb

Merge branch 'master' into AlertsMetrics

8a176a8

tests

ce0cd8a

Merge branch 'master' into AlertsMetrics

a3eecb5

Fix Tests

ba88e56

joshw123 requested review from juliocc and ludoo December 13, 2024 15:28

Merge branch 'master' into AlertsMetrics

38ace70

Fix formatting

38153c9

ludoo reviewed Dec 17, 2024

View reviewed changes

Merge remote-tracking branch 'origin/master' into AlertsMetrics

414c698

juliocc requested changes Dec 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Alerts, Logging, Channels Factories #2758

Add Alerts, Logging, Channels Factories #2758

joshw123 commented Dec 10, 2024 •

edited

Loading

juliocc left a comment •

edited

Loading

joshw123 commented Dec 11, 2024

joshw123 commented Dec 13, 2024

juliocc commented Dec 13, 2024

ludoo commented Dec 15, 2024

juliocc commented Dec 15, 2024

juliocc commented Dec 16, 2024

ludoo Dec 17, 2024

juliocc Dec 17, 2024

ludoo Dec 17, 2024

ludoo Dec 17, 2024

ludoo Dec 17, 2024

ludoo Dec 17, 2024

ludoo Dec 17, 2024

juliocc left a comment

juliocc Dec 12, 2024

juliocc Dec 18, 2024

juliocc Dec 12, 2024

ludoo Dec 18, 2024

juliocc Dec 17, 2024

juliocc Dec 18, 2024

juliocc Dec 18, 2024

Add Alerts, Logging, Channels Factories #2758

Are you sure you want to change the base?

Add Alerts, Logging, Channels Factories #2758

Conversation

joshw123 commented Dec 10, 2024 • edited Loading

juliocc left a comment • edited Loading

Choose a reason for hiding this comment

joshw123 commented Dec 11, 2024

joshw123 commented Dec 13, 2024

juliocc commented Dec 13, 2024

ludoo commented Dec 15, 2024

juliocc commented Dec 15, 2024

juliocc commented Dec 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juliocc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshw123 commented Dec 10, 2024 •

edited

Loading

juliocc left a comment •

edited

Loading