Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added job_notification_emails and deidentify actions in google_data_loss_prevention_job_trigger #7687

Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 107 additions & 0 deletions mmv1/products/dlp/JobTrigger.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,8 @@ properties:
- pub_sub
- publish_findings_to_cloud_data_catalog
- publish_summary_to_cscc
- job_notification_emails
- deidentify
description: |
If set, the detailed findings will be persisted to the specified OutputStorageConfig. Only a single instance of this action can be specified. Compatible with: Inspect, Risk
properties:
Expand Down Expand Up @@ -425,6 +427,8 @@ properties:
- pub_sub
- publish_findings_to_cloud_data_catalog
- publish_summary_to_cscc
- job_notification_emails
- deidentify
description: |
Publish a message into a given Pub/Sub topic when the job completes.
properties:
Expand All @@ -440,6 +444,8 @@ properties:
- pub_sub
- publish_findings_to_cloud_data_catalog
- publish_summary_to_cscc
- job_notification_emails
- deidentify
allow_empty_object: true
send_empty_value: true
properties: []
Expand All @@ -452,8 +458,109 @@ properties:
- pub_sub
- publish_findings_to_cloud_data_catalog
- publish_summary_to_cscc
- job_notification_emails
- deidentify
allow_empty_object: true
send_empty_value: true
properties: []
description: |
Publish findings of a DlpJob to Data Catalog.
- !ruby/object:Api::Type::NestedObject
name: 'jobNotificationEmails'
exactly_one_of:
- save_findings
- pub_sub
- publish_findings_to_cloud_data_catalog
- publish_summary_to_cscc
- job_notification_emails
- deidentify
allow_empty_object: true
send_empty_value: true
properties: []
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to double check that the value of jobNotificationEmails should be {} ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the API docs, there is no information about the object of jobNotificationEmails. Moreover, I've tried sending jobNotificationEmails as {} and it's working fine. I've also checked this with the API explorer and it is not suggesting any fields inside jobNotificationEmails.

Following is the payload I've used to configure jobTriggers with jobNotificationEmails action:

---[ REQUEST ]---------------------------------------
{
 "jobTrigger": {
  "description": "Description for the job_trigger created by terraform",
  "displayName": "TerraformDisplayName",
  "inspectJob": {
   "actions": [
    {
     "jobNotificationEmails": {},
     "publishFindingsToCloudDataCatalog": null,
     "publishSummaryToCscc": null
    }
   ],
   "inspectTemplateName": "sample-inspect-template",
   "storageConfig": {
    "cloudStorageOptions": {
     "fileSet": {
      "url": "gs://mybucket/directory/"
     }
    }
   }
  },
  "status": "HEALTHY",
  "triggers": [
   {
    "schedule": {
     "recurrencePeriodDuration": "86400s"
    }
   }
  ]
 }
}

---[ RESPONSE ]--------------------------------------
{
  "name": "projects/{project_name}/jobTriggers/{job_trigger_id}",
  "displayName": "TerraformDisplayName",
  "description": "Description for the job_trigger created by terraform",
  "inspectJob": {
    "storageConfig": {
      "cloudStorageOptions": {
        "fileSet": {
          "url": "gs://mybucket/directory/"
        }
      }
    },
    "inspectTemplateName": "sample-inspect-template",
    "actions": [
      {
        "jobNotificationEmails": {}
      }
    ]
  },
  "triggers": [
    {
      "schedule": {
        "recurrencePeriodDuration": "86400s"
      }
    }
  ],
  "createTime": "2023-04-13T08:22:50.200795Z",
  "updateTime": "2023-04-13T08:22:50.200795Z",
  "status": "HEALTHY"
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot find the doc for this field, either. I asked the service team and am waiting for their response. Sorry for holding the PR a little bit.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked with the service team and they confirmed that this field should be an empty json object.

description: |
Sends an email when the job completes. The email goes to IAM project owners and technical Essential Contacts.
- !ruby/object:Api::Type::NestedObject
name: 'deidentify'
exactly_one_of:
- save_findings
- pub_sub
- publish_findings_to_cloud_data_catalog
- publish_summary_to_cscc
- job_notification_emails
- deidentify
description: |
Create a de-identified copy of the requested table or files.
properties:
- !ruby/object:Api::Type::String
name: 'cloudStorageOutput'
required: true
description: |
User settable Cloud Storage bucket and folders to store de-identified files.

This field must be set for cloud storage deidentification.

The output Cloud Storage bucket must be different from the input bucket.

De-identified files will overwrite files in the output path.

Form of: gs://bucket/folder/ or gs://bucket
- !ruby/object:Api::Type::Array
name: 'fileTypesToTransform'
description: |
List of user-specified file type groups to transform. If specified, only the files with these filetypes will be transformed.

If empty, all supported files will be transformed. Supported types may be automatically added over time.

If a file type is set in this field that isn't supported by the Deidentify action then the job will fail and will not be successfully created/started.
item_type: !ruby/object:Api::Type::Enum
name: 'undefined'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name should be fileType?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out. I've updated the name to fileType so that it is relevant as per the API docs. Just wanted to clarify that this name won't appear in the generated provider.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can click the link in Diff report to check the generated resource in the provider.

https://github.com/modular-magician/terraform-provider-google/compare/auto-pr-7687-old..auto-pr-7687

file_type is not there.

description: |
This field only has a name and description because of MM
limitations. It should not appear in downstreams.
values:
- :IMAGES
- :TEXT_FILES
- :CSV
- :TSV
- !ruby/object:Api::Type::NestedObject
name: 'transformationConfig'
description: |
User specified deidentify templates and configs for structured, unstructured, and image files.
properties:
- !ruby/object:Api::Type::String
name: 'deidentifyTemplate'
description: |
If this template is specified, it will serve as the default de-identify template.
- !ruby/object:Api::Type::String
name: 'structuredDeidentifyTemplate'
description: |
If this template is specified, it will serve as the de-identify template for structured content such as delimited files and tables.
- !ruby/object:Api::Type::String
name: 'imageRedactTemplate'
description: |
If this template is specified, it will serve as the de-identify template for images.
- !ruby/object:Api::Type::NestedObject
name: 'transformationDetailsStorageConfig'
description: |
Config for storing transformation details.
properties:
- !ruby/object:Api::Type::NestedObject
name: 'table'
required: true
description: |
The BigQuery table in which to store the output.
properties:
- !ruby/object:Api::Type::String
name: 'datasetId'
description: The ID of the dataset containing this table.
required: true
- !ruby/object:Api::Type::String
name: 'projectId'
description: The ID of the project containing this table.
required: true
- !ruby/object:Api::Type::String
name: 'tableId'
description: |
The ID of the table. The ID must contain only letters (a-z,
A-Z), numbers (0-9), or underscores (_). The maximum length
is 1,024 characters.
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,57 @@ func TestAccDataLossPreventionJobTrigger_dlpJobTriggerPubsub(t *testing.T) {
})
}

func TestAccDataLossPreventionJobTrigger_dlpJobTriggerJobNotificationEmails(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please move these two new tests to examples in JobTrigger.yaml file? https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/products/dlp/JobTrigger.yaml#L28

In this file mmv1/third_party/terraform/tests/resource_data_loss_prevention_job_trigger_test.go, can you please add tests for updating field?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've moved these tests to examples and also added the update tests.

t.Parallel()

context := map[string]interface{}{
"project": GetTestProjectFromEnv(),
}

VcrTest(t, resource.TestCase{
PreCheck: func() { AccTestPreCheck(t) },
ProtoV5ProviderFactories: ProtoV5ProviderFactories(t),
CheckDestroy: testAccCheckDataLossPreventionJobTriggerDestroyProducer(t),
Steps: []resource.TestStep{
{
Config: testAccDataLossPreventionJobTrigger_dlpJobTriggerJobNotificationEmails(context),
},
{
ResourceName: "google_data_loss_prevention_job_trigger.job_notification_emails",
ImportState: true,
ImportStateVerify: true,
ImportStateVerifyIgnore: []string{"parent"},
},
},
})
}

func TestAccDataLossPreventionJobTrigger_dlpJobTriggerDeidentify(t *testing.T) {
t.Parallel()

context := map[string]interface{}{
"project": GetTestProjectFromEnv(),
"random_suffix": RandString(t, 10),
}

VcrTest(t, resource.TestCase{
PreCheck: func() { AccTestPreCheck(t) },
ProtoV5ProviderFactories: ProtoV5ProviderFactories(t),
CheckDestroy: testAccCheckDataLossPreventionJobTriggerDestroyProducer(t),
Steps: []resource.TestStep{
{
Config: testAccDataLossPreventionJobTrigger_dlpJobTriggerDeidentify(context),
},
{
ResourceName: "google_data_loss_prevention_job_trigger.deidentify",
ImportState: true,
ImportStateVerify: true,
ImportStateVerifyIgnore: []string{"parent"},
},
},
})
}

func testAccDataLossPreventionJobTrigger_dlpJobTriggerBasic(context map[string]interface{}) string {
return Nprintf(`
resource "google_data_loss_prevention_job_trigger" "basic" {
Expand Down Expand Up @@ -294,3 +345,121 @@ resource "google_data_loss_prevention_job_trigger" "pubsub" {
}
`, context)
}

func testAccDataLossPreventionJobTrigger_dlpJobTriggerJobNotificationEmails(context map[string]interface{}) string {
return Nprintf(`
resource "google_data_loss_prevention_job_trigger" "job_notification_emails" {
parent = "projects/%{project}"
description = "Description for the job_trigger created by terraform"
display_name = "TerraformDisplayName"

triggers {
schedule {
recurrence_period_duration = "86400s"
}
}

inspect_job {
inspect_template_name = "sample-inspect-template"
actions {
job_notification_emails {}
}
storage_config {
cloud_storage_options {
file_set {
url = "gs://mybucket/directory/"
}
}
}
}
}
`, context)
}

func testAccDataLossPreventionJobTrigger_dlpJobTriggerDeidentify(context map[string]interface{}) string {
return Nprintf(`
resource "google_data_loss_prevention_job_trigger" "deidentify" {
parent = "projects/%{project}"
description = "Description for the job_trigger created by terraform"
display_name = "TerraformDisplayName"

triggers {
schedule {
recurrence_period_duration = "86400s"
}
}

inspect_job {
inspect_template_name = "sample-inspect-template"
actions {
deidentify {
cloud_storage_output = "gs://samplebucket/dir/"
file_types_to_transform = ["CSV", "TSV"]
transformation_details_storage_config {
table {
project_id = "%{project}"
dataset_id = google_bigquery_dataset.default.dataset_id
table_id = google_bigquery_table.default.table_id
}
}
transformation_config {
deidentify_template = "sample-deidentify-template"
image_redact_template = "sample-image-redact-template"
structured_deidentify_template = "sample-structured-deidentify-template"
}
}
}
storage_config {
cloud_storage_options {
file_set {
url = "gs://mybucket/directory/"
}
}
}
}
}

resource "google_bigquery_dataset" "default" {
dataset_id = "tf_test_%{random_suffix}"
friendly_name = "terraform-test"
description = "Description for the dataset created by terraform"
location = "US"
default_table_expiration_ms = 3600000

labels = {
env = "default"
}
}

resource "google_bigquery_table" "default" {
dataset_id = google_bigquery_dataset.default.dataset_id
table_id = "tf_test_%{random_suffix}"
deletion_protection = false

time_partitioning {
type = "DAY"
}

labels = {
env = "default"
}

schema = <<EOF
[
{
"name": "quantity",
"type": "NUMERIC",
"mode": "NULLABLE",
"description": "The quantity"
},
{
"name": "name",
"type": "STRING",
"mode": "NULLABLE",
"description": "Name of the object"
}
]
EOF
}
`, context)
}