Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

inspectConfig added to jobTrigger #7677

Merged

Conversation

JayS-crest
Copy link
Contributor

Added inspectConfig block with all the attributes to inspectJob in jobTrigger resource and also added relevant test cases.

If this PR is for Terraform, I acknowledge that I have:

  • Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
  • Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
  • Generated Terraform providers, and ran make test and make lint in the generated providers to ensure it passes unit and linter tests.
  • Ran relevant acceptance tests using my own Google Cloud project and credentials (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
  • Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

dlp: added `inspectConfig` field to `google_data_loss_prevention_job_trigger` resource

@JayS-crest JayS-crest requested a review from a team as a code owner April 10, 2023 05:42
@JayS-crest JayS-crest requested review from slevenick and removed request for a team April 10, 2023 05:42
@modular-magician modular-magician added the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Apr 10, 2023
@JayS-crest
Copy link
Contributor Author

@slevenick Can you start the tests? It is awaiting approval from your end, Thanks.

@zli82016 zli82016 requested review from zli82016 and removed request for slevenick April 13, 2023 16:41
@modular-magician modular-magician removed the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Apr 13, 2023
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 4153 insertions(+), 201 deletions(-))
Terraform Beta: Diff ( 4 files changed, 4153 insertions(+), 201 deletions(-))
TF Conversion: Diff ( 3 files changed, 1201 insertions(+), 3 deletions(-))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_data_loss_prevention_job_trigger (14 total tests)
Untested fields: inspect_job.inspect_config.rule_set.rules.hotword_rule.hotword_regex.pattern, inspect_job.inspect_config.rule_set.rules.hotword_rule.hotword_regex.group_indexes, inspect_job.inspect_config.rule_set.rules.hotword_rule.likelihood_adjustment.fixed_likelihood, inspect_job.inspect_config.rule_set.rules.hotword_rule.likelihood_adjustment.relative_likelihood, inspect_job.inspect_config.rule_set.rules.hotword_rule.proximity.window_after, inspect_job.inspect_config.rule_set.rules.hotword_rule.proximity.window_before, inspect_job.inspect_config.rule_set.rules.exclusion_rule.regex.group_indexes, inspect_job.inspect_config.rule_set.rules.exclusion_rule.dictionary.cloud_storage_path.path, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_by_hotword.hotword_regex.pattern, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_by_hotword.hotword_regex.group_indexes, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_by_hotword.proximity.window_after, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_by_hotword.proximity.window_before, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_info_types.info_types.version, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_info_types.info_types.name, inspect_job.inspect_config.rule_set.info_types.version, inspect_job.inspect_config.content_options, inspect_job.inspect_config.custom_info_types.detection_rules.hotword_rule.hotword_regex.pattern, inspect_job.inspect_config.custom_info_types.detection_rules.hotword_rule.hotword_regex.group_indexes, inspect_job.inspect_config.custom_info_types.detection_rules.hotword_rule.likelihood_adjustment.fixed_likelihood, inspect_job.inspect_config.custom_info_types.detection_rules.hotword_rule.likelihood_adjustment.relative_likelihood, inspect_job.inspect_config.custom_info_types.detection_rules.hotword_rule.proximity.window_before, inspect_job.inspect_config.custom_info_types.detection_rules.hotword_rule.proximity.window_after, inspect_job.inspect_config.custom_info_types.dictionary.cloud_storage_path.path, inspect_job.inspect_config.custom_info_types.dictionary.word_list.words, inspect_job.inspect_config.custom_info_types.exclusion_type, inspect_job.inspect_config.custom_info_types.regex.group_indexes, inspect_job.inspect_config.custom_info_types.stored_type.name, inspect_job.inspect_config.custom_info_types.info_type.version, inspect_job.inspect_config.exclude_info_types, inspect_job.inspect_config.include_quote, inspect_job.inspect_config.limits.max_findings_per_info_type.info_type.version

Please add acceptance tests which include these fields.

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 0
Passed tests 0
Skipped tests: 0
Affected tests: 0

Errors occurred during REPLAYING mode. Please fix them to complete your PR
View the build log

@JayS-crest
Copy link
Contributor Author

JayS-crest commented Apr 13, 2023

Is inclusion of all the added fields in tests is necessary? If that's the case, will add a few more tests which cover all the attributes. Also, I've noticed some fields which I've added to tests are also showed missing. For example, inspect_job.inspect_config.rule_set.rules.hotword_rule.hotword_regex.pattern is missing as per above comment, but here it is present in the basic test. Any suggestions?

}

VcrTest(t, resource.TestCase{
PreCheck: func() { testAccPreCheck(t) },
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function has to be AccTestPreCheck

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the value. And to me, it seems like PreCheck used to fail and gave error for missing tests for certain attributes.

@zli82016
Copy link
Member

Is inclusion of all the added fields in tests is necessary? If that's the case, will add a few more tests which cover all the attributes. Also, I've noticed some fields which I've added to tests are also showed missing. For example, inspect_job.inspect_config.rule_set.rules.hotword_rule.hotword_regex.pattern is missing as per above comment, but here it is present in the basic test. Any suggestions?

It is recommended to include all fields in tests. If the test has been added for a field, you can ignore the note for this field in the Missing test report.

@modular-magician modular-magician added the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Apr 14, 2023
@JayS-crest
Copy link
Contributor Author

JayS-crest commented Apr 17, 2023

@zli82016 I've added extra tests and done the necessary changes. Can you please start the tests?

@modular-magician modular-magician removed the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Apr 17, 2023
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 4267 insertions(+), 337 deletions(-))
Terraform Beta: Diff ( 4 files changed, 4267 insertions(+), 337 deletions(-))
TF Conversion: Diff ( 3 files changed, 1026 insertions(+), 3 deletions(-))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_data_loss_prevention_job_trigger (26 total tests)
Untested fields: inspect_job.inspect_config.limits.max_findings_per_info_type.info_type.version, inspect_job.inspect_config.rule_set.info_types.version, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_info_types.info_types.name, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_info_types.info_types.version, inspect_job.inspect_config.rule_set.rules.exclusion_rule.regex.group_indexes, inspect_job.inspect_config.rule_set.rules.exclusion_rule.dictionary.cloud_storage_path.path, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_by_hotword.proximity.window_after, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_by_hotword.proximity.window_before, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_by_hotword.hotword_regex.pattern, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_by_hotword.hotword_regex.group_indexes, inspect_job.inspect_config.rule_set.rules.hotword_rule.hotword_regex.pattern, inspect_job.inspect_config.rule_set.rules.hotword_rule.hotword_regex.group_indexes, inspect_job.inspect_config.rule_set.rules.hotword_rule.likelihood_adjustment.fixed_likelihood, inspect_job.inspect_config.rule_set.rules.hotword_rule.likelihood_adjustment.relative_likelihood, inspect_job.inspect_config.rule_set.rules.hotword_rule.proximity.window_before, inspect_job.inspect_config.rule_set.rules.hotword_rule.proximity.window_after, inspect_job.inspect_config.custom_info_types.regex.group_indexes, inspect_job.inspect_config.custom_info_types.info_type.version, inspect_job.inspect_config.custom_info_types.dictionary.cloud_storage_path.path

Please add acceptance tests which include these fields.

- !ruby/object:Api::Type::Boolean
name: 'excludeInfoTypes'
description: When true, excludes type information of the findings.
- !ruby/object:Api::Type::Boolean
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To send the false value, send_empty_value: true is needed for this field.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this field a regular boolean value which can be sent true/false without send_empty_value: true? I found the same field in resource dlp_inspect_template in a similar way here. Similar to above declared attribute excludeInfoTypes .

name: 'maxFindingsPerItem'
description: Max number of findings that will be returned for each item
scanned. The maximum returned is 2000.
required: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to double check if this field is required.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the field manually and it isn't required, same goes for all other fields in limits. Hence, I've added AtleastOneOf instead of keeping all of them required.

@zli82016
Copy link
Member

I've done all the changes, removed immutable: true as well. I've also added missing attributes to test and I guess that should probably work. Please run the tests and review at your convenience.

I've also added extra cases covering most of the attributes in jobTrigger but found out many were still showing in Missing Test Report. I don't know the reason and as you had mentioned earlier, I've ignored that. Thanks!

Thanks so much for doing the check and making the changes.

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 4792 insertions(+), 338 deletions(-))
Terraform Beta: Diff ( 4 files changed, 4792 insertions(+), 338 deletions(-))
TF Conversion: Diff ( 3 files changed, 1026 insertions(+), 3 deletions(-))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_data_loss_prevention_job_trigger (29 total tests)
Untested fields: inspect_job.inspect_config.rule_set.rules.hotword_rule.proximity.window_after, inspect_job.inspect_config.rule_set.rules.hotword_rule.proximity.window_before, inspect_job.inspect_config.rule_set.rules.hotword_rule.hotword_regex.pattern, inspect_job.inspect_config.rule_set.rules.hotword_rule.hotword_regex.group_indexes, inspect_job.inspect_config.rule_set.rules.hotword_rule.likelihood_adjustment.fixed_likelihood, inspect_job.inspect_config.rule_set.rules.hotword_rule.likelihood_adjustment.relative_likelihood, inspect_job.inspect_config.rule_set.rules.exclusion_rule.dictionary.cloud_storage_path.path, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_by_hotword.hotword_regex.group_indexes, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_by_hotword.hotword_regex.pattern, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_by_hotword.proximity.window_after, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_by_hotword.proximity.window_before, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_info_types.info_types.name, inspect_job.inspect_config.rule_set.rules.exclusion_rule.exclude_info_types.info_types.version, inspect_job.inspect_config.rule_set.rules.exclusion_rule.regex.group_indexes

Please add acceptance tests which include these fields.

@JayS-crest
Copy link
Contributor Author

I've added the missing fields, still it's being shown in the Missing Test Report. Any reasons for this to happen?

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2646
Passed tests 2361
Skipped tests: 277
Affected tests: 8

Action taken

Found 8 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccDataLossPreventionJobTrigger_dlpJobTriggerInspectCustomInfoTypes|TestAccApigeeKeystoresAliasesPkcs12_ApigeeKeystoresAliasesPkcs12Example|TestAccApigeeKeystoresAliasesKeyCertFile_apigeeKeystoresAliasesKeyCertFileTestExample|TestAccAlloydbCluster_missingLocation|TestAccAlloydbBackup_missingLocation|TestAccComputeForwardingRule_update|TestAccDataSourceAlloydbLocations_basic|TestAccDataSourceGoogleFirebaseAndroidAppConfig

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccAlloydbCluster_missingLocation[Debug log]
TestAccAlloydbBackup_missingLocation[Debug log]
TestAccComputeForwardingRule_update[Debug log]
TestAccDataSourceAlloydbLocations_basic[Debug log]
TestAccDataSourceGoogleFirebaseAndroidAppConfig[Debug log]

Tests failed during RECORDING mode:
TestAccDataLossPreventionJobTrigger_dlpJobTriggerInspectCustomInfoTypes[Error message] [Debug log]
TestAccApigeeKeystoresAliasesPkcs12_ApigeeKeystoresAliasesPkcs12Example[Error message] [Debug log]
TestAccApigeeKeystoresAliasesKeyCertFile_apigeeKeystoresAliasesKeyCertFileTestExample[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

}
proximity {
window_before = 25
windows_after = 25
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be window_after

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to window_after.

@zli82016
Copy link
Member

I've added the missing fields, still it's being shown in the Missing Test Report. Any reasons for this to happen?

I am not sure and going to ask my team.

@trodge
Copy link
Contributor

trodge commented Apr 20, 2023

I've added the missing fields, still it's being shown in the Missing Test Report. Any reasons for this to happen?

There was an issue with the missing test detector which this PR should fix.

Here is the report after applying the fix:

Resource: google_data_loss_prevention_job_trigger (29 total tests)
Untested fields: inspect_job.inspect_config.rule_set.rules.hotword_rule.hotword_regex.group_indexes, inspect_job.inspect_config.rule_set.rules.hotword_rule.likelihood_adjustment.relative_likelihood, inspect_job.inspect_config.rule_set.rules.hotword_rule.proximity.window_after, inspect_job.inspect_config.rule_set.rules.exclusion_rule.regex.group_indexes

@JayS-crest
Copy link
Contributor Author

There was an issue with the missing test detector which this PR should fix.

Here is the report after applying the fix:

Resource: google_data_loss_prevention_job_trigger (29 total tests) Untested fields: inspect_job.inspect_config.rule_set.rules.hotword_rule.hotword_regex.group_indexes, inspect_job.inspect_config.rule_set.rules.hotword_rule.likelihood_adjustment.relative_likelihood, inspect_job.inspect_config.rule_set.rules.hotword_rule.proximity.window_after, inspect_job.inspect_config.rule_set.rules.exclusion_rule.regex.group_indexes

Thanks for the help, @trodge. I've added the remaining fields to the config.

@modular-magician modular-magician added the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Apr 21, 2023
@modular-magician modular-magician removed the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Apr 21, 2023
@zli82016
Copy link
Member

/gcbrun

@modular-magician modular-magician added awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests and removed awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests labels Apr 21, 2023
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 4794 insertions(+), 338 deletions(-))
Terraform Beta: Diff ( 4 files changed, 4794 insertions(+), 338 deletions(-))
TF Conversion: Diff ( 3 files changed, 1026 insertions(+), 3 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2665
Passed tests 2377
Skipped tests: 280
Affected tests: 8

Action taken

Found 8 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccFirebaserulesRelease_BasicRelease|TestAccComputeForwardingRule_update|TestAccAlloydbCluster_missingLocation|TestAccApigeeKeystoresAliasesKeyCertFile_apigeeKeystoresAliasesKeyCertFileTestExample|TestAccApigeeKeystoresAliasesPkcs12_ApigeeKeystoresAliasesPkcs12Example|TestAccAlloydbBackup_missingLocation|TestAccDataSourceAlloydbLocations_basic|TestAccDataSourceGoogleFirebaseAndroidAppConfig

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccComputeForwardingRule_update[Debug log]
TestAccAlloydbCluster_missingLocation[Debug log]
TestAccApigeeKeystoresAliasesKeyCertFile_apigeeKeystoresAliasesKeyCertFileTestExample[Debug log]
TestAccApigeeKeystoresAliasesPkcs12_ApigeeKeystoresAliasesPkcs12Example[Debug log]
TestAccAlloydbBackup_missingLocation[Debug log]
TestAccDataSourceAlloydbLocations_basic[Debug log]

Tests failed during RECORDING mode:
TestAccDataSourceGoogleFirebaseAndroidAppConfig[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

Copy link
Member

@zli82016 zli82016 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks.

@zli82016 zli82016 merged commit 4f6a2e7 into GoogleCloudPlatform:main Apr 21, 2023
ravisiddhu pushed a commit to ravisiddhu/magic-modules that referenced this pull request Apr 25, 2023
* inspectConfig added to jobTrigger

* Test Precheck Updated

* fields added to test

* removed deprecated attributes contentOptions and detectionRules

* customInfoTypes tests added covering all attributes

* tabs removed from erb file and double space added

* removed required from mentioned attributes

* minor changes

* Additional test for Exclusion Rule added

* removing required from hotwordRegex and pattern

* Spacing changed

* immutable removed from customInfoType attributes

* required removed from proximity

* required removed from proximity and likelihoodAdjustment in rules

* Tests modified and new cases added

* added remaining 4 attr to test config

* minor changes
maphad pushed a commit to maphad/magic-modules that referenced this pull request Apr 27, 2023
* inspectConfig added to jobTrigger

* Test Precheck Updated

* fields added to test

* removed deprecated attributes contentOptions and detectionRules

* customInfoTypes tests added covering all attributes

* tabs removed from erb file and double space added

* removed required from mentioned attributes

* minor changes

* Additional test for Exclusion Rule added

* removing required from hotwordRegex and pattern

* Spacing changed

* immutable removed from customInfoType attributes

* required removed from proximity

* required removed from proximity and likelihoodAdjustment in rules

* Tests modified and new cases added

* added remaining 4 attr to test config

* minor changes
maphad pushed a commit to maphad/magic-modules that referenced this pull request Apr 27, 2023
* inspectConfig added to jobTrigger

* Test Precheck Updated

* fields added to test

* removed deprecated attributes contentOptions and detectionRules

* customInfoTypes tests added covering all attributes

* tabs removed from erb file and double space added

* removed required from mentioned attributes

* minor changes

* Additional test for Exclusion Rule added

* removing required from hotwordRegex and pattern

* Spacing changed

* immutable removed from customInfoType attributes

* required removed from proximity

* required removed from proximity and likelihoodAdjustment in rules

* Tests modified and new cases added

* added remaining 4 attr to test config

* minor changes
maphad pushed a commit to maphad/magic-modules that referenced this pull request Apr 27, 2023
* inspectConfig added to jobTrigger

* Test Precheck Updated

* fields added to test

* removed deprecated attributes contentOptions and detectionRules

* customInfoTypes tests added covering all attributes

* tabs removed from erb file and double space added

* removed required from mentioned attributes

* minor changes

* Additional test for Exclusion Rule added

* removing required from hotwordRegex and pattern

* Spacing changed

* immutable removed from customInfoType attributes

* required removed from proximity

* required removed from proximity and likelihoodAdjustment in rules

* Tests modified and new cases added

* added remaining 4 attr to test config

* minor changes
ericayyliu pushed a commit to ericayyliu/magic-modules that referenced this pull request Jul 26, 2023
* inspectConfig added to jobTrigger

* Test Precheck Updated

* fields added to test

* removed deprecated attributes contentOptions and detectionRules

* customInfoTypes tests added covering all attributes

* tabs removed from erb file and double space added

* removed required from mentioned attributes

* minor changes

* Additional test for Exclusion Rule added

* removing required from hotwordRegex and pattern

* Spacing changed

* immutable removed from customInfoType attributes

* required removed from proximity

* required removed from proximity and likelihoodAdjustment in rules

* Tests modified and new cases added

* added remaining 4 attr to test config

* minor changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants