Add a resource: Discovery Engine Data Store #9860

jialei-chen · 2024-01-22T23:12:29Z

`google_discovery_engine_data_store`

…I resource's documented behavior

…bility improvement.

…enerated resource name would be google_discoveryengine_data_store

Co-authored-by: Edward Sun <[email protected]>

…printf format %s reads arg GoogleCloudPlatform#3, but call has 2 args'.

…form#9536

… industry_vertical, content_config.

…field of CreateDataStoreRequest.

… import_format.

… remove fields for std Update method; (3) rename structured datastore example.

…irements.

modular-magician · 2024-01-22T23:13:51Z

Hello! I am a robot. It looks like you are a: ~~Community Contributor~~ Googler ~~Core Contributor~~. Tests will run automatically.

@rileykarson, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

jialei-chen · 2024-01-22T23:42:35Z

@roaks3 Hi Ryan, it seems I cannot directly modify the reviewer list to add you there. So here is a friendly reminder.

Given the context, I finished as our discussion. Here is a brief summary: all codes remain the same as that in #9682 except for the following two changes:

Updated API version from v1alpha to v1.
Supported regional APIs: from discoveryengine.googleapis.com to {{location}}-discoveryengine.googleapis.com.

Thanks!

modular-magician · 2024-01-23T00:19:28Z

Tests analytics

Total tests: 3374
Passed tests 3004
Skipped tests: 337
Affected tests: 33

Click here to see the affected service packages

all service packages are affected

Action taken

Found 33 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccProviderMeta_setModuleName|TestAccAccessContextManager|TestAccAlloydbCluster_alloydbSecondaryClusterBasicTestExample|TestAccAlloydbUser_alloydbUserIamTestExample|TestAccApigeeEnvReferences_apigeeEnvironmentReferenceTestExample|TestAccCertificateManagerTrustConfig_certificateManagerTrustConfigExample|TestAccCloudfunctions2function_cloudfunctions2BasicAuditlogsExample|TestAccCloudIdentityGroup|TestAccContainerCluster_backend|TestAccDiscoveryEngineDataStore_discoveryengineDatastoreBasicExample_update|TestAccDiscoveryEngineDataStore_discoveryengineDatastoreBasicExample|TestAccDNSRecordSet_routingPolicy|TestAccIapWebBackendServiceIamMemberGenerated_withCondition|TestAccIapWebBackendServiceIamBindingGenerated_withCondition|TestAccIapWebBackendServiceIamBindingGenerated|TestAccIapWebBackendServiceIamMemberGenerated|TestAccIapWebRegionBackendServiceIamPolicyGenerated_withCondition|TestAccIapWebBackendServiceIamPolicyGenerated|TestAccIapWebRegionBackendServiceIamMemberGenerated_withCondition|TestAccIapWebRegionBackendServiceIamBindingGenerated_withCondition|TestAccIapWebRegionBackendServiceIamPolicyGenerated|TestAccIapWebRegionBackendServiceIamMemberGenerated|TestAccIapWebRegionBackendServiceIamBindingGenerated|TestAccIapWebBackendServiceIamPolicyGenerated_withCondition|TestAccNetworkServicesTcpRoute_update|TestAccNetworkServicesTcpRoute_networkServicesTcpRouteGatewayBasicExample|TestAccNetworkServicesTcpRoute_networkServicesTcpRouteMeshBasicExample|TestAccNetworkServicesTcpRoute_networkServicesTcpRouteActionsExample|TestAccNetworkServicesTcpRoute_networkServicesTcpRouteBasicExample|TestAccNetworkServicesTlsRoute_networkServicesTlsRouteGatewayBasicExample|TestAccNetworkServicesTlsRoute_update|TestAccNetworkServicesTlsRoute_networkServicesTlsRouteMeshBasicExample|TestAccNetworkServicesTlsRoute_networkServicesTlsRouteBasicExample

Get to know how VCR tests work

modular-magician · 2024-01-23T01:30:20Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccProviderMeta_setModuleName[Debug log]
TestAccAccessContextManager[Debug log]
TestAccAlloydbCluster_alloydbSecondaryClusterBasicTestExample[Debug log]
TestAccAlloydbUser_alloydbUserIamTestExample[Debug log]
TestAccApigeeEnvReferences_apigeeEnvironmentReferenceTestExample[Debug log]
TestAccCertificateManagerTrustConfig_certificateManagerTrustConfigExample[Debug log]
TestAccCloudfunctions2function_cloudfunctions2BasicAuditlogsExample[Debug log]
TestAccCloudIdentityGroup[Debug log]
TestAccContainerCluster_backend[Debug log]
TestAccDiscoveryEngineDataStore_discoveryengineDatastoreBasicExample_update[Debug log]
TestAccDiscoveryEngineDataStore_discoveryengineDatastoreBasicExample[Debug log]
TestAccDNSRecordSet_routingPolicy[Debug log]
TestAccIapWebBackendServiceIamMemberGenerated_withCondition[Debug log]
TestAccIapWebBackendServiceIamBindingGenerated_withCondition[Debug log]
TestAccIapWebBackendServiceIamBindingGenerated[Debug log]
TestAccIapWebBackendServiceIamMemberGenerated[Debug log]
TestAccIapWebRegionBackendServiceIamPolicyGenerated_withCondition[Debug log]
TestAccIapWebBackendServiceIamPolicyGenerated[Debug log]
TestAccIapWebRegionBackendServiceIamMemberGenerated_withCondition[Debug log]
TestAccIapWebRegionBackendServiceIamBindingGenerated_withCondition[Debug log]
TestAccIapWebRegionBackendServiceIamPolicyGenerated[Debug log]
TestAccIapWebRegionBackendServiceIamMemberGenerated[Debug log]
TestAccIapWebRegionBackendServiceIamBindingGenerated[Debug log]
TestAccIapWebBackendServiceIamPolicyGenerated_withCondition[Debug log]
TestAccNetworkServicesTcpRoute_update[Debug log]
TestAccNetworkServicesTcpRoute_networkServicesTcpRouteGatewayBasicExample[Debug log]
TestAccNetworkServicesTcpRoute_networkServicesTcpRouteMeshBasicExample[Debug log]
TestAccNetworkServicesTcpRoute_networkServicesTcpRouteActionsExample[Debug log]
TestAccNetworkServicesTcpRoute_networkServicesTcpRouteBasicExample[Debug log]
TestAccNetworkServicesTlsRoute_update[Debug log]
TestAccNetworkServicesTlsRoute_networkServicesTlsRouteMeshBasicExample[Debug log]
TestAccNetworkServicesTlsRoute_networkServicesTlsRouteBasicExample[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccNetworkServicesTlsRoute_networkServicesTlsRouteGatewayBasicExample[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

modular-magician · 2024-01-23T01:43:38Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 14 files changed, 1094 insertions(+), 2 deletions(-))
Terraform Beta: Diff ( 14 files changed, 1094 insertions(+), 2 deletions(-))
TF Conversion: Diff ( 1 file changed, 99 insertions(+))

Pratster95 · 2024-01-23T02:23:49Z

mmv1/products/discoveryengine/DataStore.yaml

+references: !ruby/object:Api::Resource::ReferenceLinks
+  guides:
+    'Create a search data store': 'https://cloud.google.com/generative-ai-app-builder/docs/create-data-store-es'
+  api: 'https://cloud.google.com/generative-ai-app-builder/docs/reference/rest/v1alpha/projects.locations.collections.dataStores'


@jialei-chen The reference doc points to alpha API

Thanks. Updated.

modular-magician · 2024-01-23T02:43:01Z

Tests analytics

Total tests: 3374
Passed tests 3036
Skipped tests: 337
Affected tests: 1

Click here to see the affected service packages

all service packages are affected

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccNetworkServicesTlsRoute_networkServicesTlsRouteGatewayBasicExample

Get to know how VCR tests work

modular-magician · 2024-01-23T02:46:52Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccNetworkServicesTlsRoute_networkServicesTlsRouteGatewayBasicExample[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

$\textcolor{green}{\textsf{All tests passed!}}$
View the build log or the debug log for each test

modular-magician · 2024-01-23T05:07:13Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 14 files changed, 1094 insertions(+), 2 deletions(-))
Terraform Beta: Diff ( 14 files changed, 1094 insertions(+), 2 deletions(-))
TF Conversion: Diff ( 1 file changed, 99 insertions(+))

modular-magician · 2024-01-23T06:07:13Z

Tests analytics

Total tests: 3374
Passed tests 3037
Skipped tests: 337
Affected tests: 0

Click here to see the affected service packages

all service packages are affected

$\textcolor{green}{\textsf{All tests passed in REPLAYING mode.}}$
View the build log

roaks3

LGTM, and things match what was previously merged.

The new location-based subdomain LGTM too, but it has been a while since we've added one of these, and I'm unsure if there are other implications. Deferring to @rileykarson to make sure that piece looks alright.

rileykarson

Nothing seems off w/ it. We'll need to make sure location is always required in the product or we ensure that parent fields include it if it isn't, as the errors where it's missing are unintuitive. No impact here, where that's not a concern yet.

@roaks3 leaving merge to you as I've looked at probably 10 lines in this PR.

roaks3 · 2024-01-23T20:50:29Z

Just did one last spot check and things looked good. Merging now.

* Create a configuration file product.yaml for product discoveryengine * Create resource DataStore configuration file template DataStore.yaml * Modify resource DataStore configuration file template to match the API resource's documented behavior * Add each API resource field to configuration file properties attribute * Add IAM support in Terraform to match DataStore API resource IAM support * Polish field contents in product configuration file. * Modify DataStore.yaml for LRO handling, parameters matching and readability improvement. * Add a DataStore create test. * Modify product name from DiscoveryEngine to Discoveryengine so that generated resource name would be google_discoveryengine_data_store * change to GA in doc (GoogleCloudPlatform#9491) Co-authored-by: Edward Sun <[email protected]> * Add value of to template into the generated tests to fix error 'fmt.Sprintf format %s reads arg GoogleCloudPlatform#3, but call has 2 args'. * Remove Api::Product::ApiReference to match the change GoogleCloudPlatform#9536 * Add var collection_id to unit test examples in resource config. * Fix argument missing error in unit tests. * Update resource create unit test case fields to contain display_name, industry_vertical, content_config. * Fix errors in acceptance tests: now acc test fails at invalid parent field of CreateDataStoreRequest. * Set collection_id as default_collection. * Remove uncessary variables under unit test example. * Fix acceptance tests issues by (1) removing iam_policy and (2) adding import_format. * Clean up configs: (1) update api version for tpg-beta to v1alpha; (2) remove fields for std Update method; (3) rename structured datastore example. * Add examples and tests for all types of DataStore. * Remove redundant field project from create test examples. * Fix lint-yaml check failure. * Add field solution_types to acceptance tests to satisfy coverage requirements. * Clean up industry_vertical enum values. * Add a new query parameter createAdvancedSiteSearch to create_url. * Add a create test for advanced site search data store. * Add IAM roles/discoveryengine.admin. * Remove role config from .ci/infra/terraform/main.tf * Remove beta version in product.yaml because both ga and beta versions have the same base_url. * Rename product from Discoveryengine to DiscoveryEngine. * Remove unnecessary configs and improve description readability. * Mark property solution_types as immutable. * Add update related settings and update test. * Add delete_url for resource's Delete method. * Remove all but one test example. * Rename test example to make it more generic. * Modify first template in update test to omit all non-required fields. * Fix the update test: update display_name not data_store_id. * Modify property description: remove brackets. * Update API version from v1alpha to v1. * Support regional APIs. * Modify API reference link. --------- Co-authored-by: Edward Sun <[email protected]> Co-authored-by: Edward Sun <[email protected]>

jialei-chen and others added 30 commits November 29, 2023 23:45

Create a configuration file product.yaml for product discoveryengine

55ef153

Create resource DataStore configuration file template DataStore.yaml

c9928b9

Modify resource DataStore configuration file template to match the AP…

d090272

…I resource's documented behavior

Add each API resource field to configuration file properties attribute

6e0aeff

Add IAM support in Terraform to match DataStore API resource IAM support

58ae514

Polish field contents in product configuration file.

45b5215

Modify DataStore.yaml for LRO handling, parameters matching and reada…

0741fa4

…bility improvement.

Add a DataStore create test.

db93751

Modify product name from DiscoveryEngine to Discoveryengine so that g…

3aa31ee

…enerated resource name would be google_discoveryengine_data_store

change to GA in doc (GoogleCloudPlatform#9491)

005dd1d

Co-authored-by: Edward Sun <[email protected]>

Add value of to template into the generated tests to fix error 'fmt.S…

451b7a1

…printf format %s reads arg GoogleCloudPlatform#3, but call has 2 args'.

Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource

5a3cdef

Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource

c730698

Remove Api::Product::ApiReference to match the change GoogleCloudPlat…

55c3d08

…form#9536

Add var collection_id to unit test examples in resource config.

4c59446

Fix argument missing error in unit tests.

f400ae3

Update resource create unit test case fields to contain display_name,…

e629831

… industry_vertical, content_config.

Fix errors in acceptance tests: now acc test fails at invalid parent …

3a73249

…field of CreateDataStoreRequest.

Set collection_id as default_collection.

c6ec24b

Remove uncessary variables under unit test example.

e3b0ac8

Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource

75a7abc

Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource

f605bac

Fix acceptance tests issues by (1) removing iam_policy and (2) adding…

a7e591f

… import_format.

Clean up configs: (1) update api version for tpg-beta to v1alpha; (2)…

a6f6f17

… remove fields for std Update method; (3) rename structured datastore example.

Add examples and tests for all types of DataStore.

fb879e9

Remove redundant field project from create test examples.

0ecd750

Fix lint-yaml check failure.

7ac2058

Add field solution_types to acceptance tests to satisfy coverage requ…

e28ad41

…irements.

Clean up industry_vertical enum values.

14535d0

Add a new query parameter createAdvancedSiteSearch to create_url.

5196ef6

jialei-chen added 2 commits January 22, 2024 22:35

Update API version from v1alpha to v1.

cc83a85

Support regional APIs.

149414a

modular-magician requested a review from rileykarson January 22, 2024 23:13

jialei-chen marked this pull request as draft January 22, 2024 23:19

jialei-chen marked this pull request as ready for review January 22, 2024 23:20

Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource

10f06cd

Pratster95 reviewed Jan 23, 2024

View reviewed changes

Modify API reference link.

850951f

modular-magician added the service/discoveryengine label Jan 23, 2024

roaks3 approved these changes Jan 23, 2024

View reviewed changes

5fff mentioned this pull request Jan 23, 2024

Add Support for Vertex Search and Conversation Chat Engine #9834

Merged

rileykarson approved these changes Jan 23, 2024

View reviewed changes

roaks3 merged commit 22bc30c into GoogleCloudPlatform:main Jan 23, 2024
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a resource: Discovery Engine Data Store #9860

Add a resource: Discovery Engine Data Store #9860

jialei-chen commented Jan 22, 2024

modular-magician commented Jan 22, 2024

jialei-chen commented Jan 22, 2024

modular-magician commented Jan 23, 2024

modular-magician commented Jan 23, 2024

modular-magician commented Jan 23, 2024

Pratster95 Jan 23, 2024

jialei-chen Jan 23, 2024

modular-magician commented Jan 23, 2024

modular-magician commented Jan 23, 2024

modular-magician commented Jan 23, 2024

modular-magician commented Jan 23, 2024

roaks3 left a comment

rileykarson left a comment

roaks3 commented Jan 23, 2024

Add a resource: Discovery Engine Data Store #9860

Add a resource: Discovery Engine Data Store #9860

Conversation

jialei-chen commented Jan 22, 2024

modular-magician commented Jan 22, 2024

jialei-chen commented Jan 22, 2024

modular-magician commented Jan 23, 2024

Tests analytics

Action taken

modular-magician commented Jan 23, 2024

Rerun these tests in REPLAYING mode to catch issues

modular-magician commented Jan 23, 2024

Diff report

Pratster95 Jan 23, 2024

Choose a reason for hiding this comment

jialei-chen Jan 23, 2024

Choose a reason for hiding this comment

modular-magician commented Jan 23, 2024

Tests analytics

Action taken

modular-magician commented Jan 23, 2024

Rerun these tests in REPLAYING mode to catch issues

modular-magician commented Jan 23, 2024

Diff report

modular-magician commented Jan 23, 2024

Tests analytics

roaks3 left a comment

Choose a reason for hiding this comment

rileykarson left a comment

Choose a reason for hiding this comment

roaks3 commented Jan 23, 2024