chore: add scripts to setup and teardown test resources (googleapis#137)

* chore: update .repo-metadata.json (googleapis#125)

* chore: use python-samples-reviewers (googleapis#128)

* chore: use gapic-generator-python 0.58.4 (googleapis#127)

* chore: use gapic-generator-python 0.58.4

fix: provide appropriate mock values for message body fields

committer: dovs
PiperOrigin-RevId: 419025932

Source-Link: googleapis/googleapis@73da669

Source-Link: https://github.com/googleapis/googleapis-gen/commit/46df624a54b9ed47c1a7eefb7a49413cf7b82f98
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNDZkZjYyNGE1NGI5ZWQ0N2MxYTdlZWZiN2E0OTQxM2NmN2I4MmY5OCJ9

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

* chore(python): update release.sh to use keystore (googleapis#130)

build: switch to release-please for tagging

* chore(main): release 1.3.0 (googleapis#131)

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

* ci(python): run lint / unit tests / docs as GH actions (googleapis#132)

* ci(python): run lint / unit tests / docs as GH actions

Source-Link: googleapis/synthtool@57be0cd
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:ed1f9983d5a935a89fe8085e8bb97d94e41015252c5b6c9771257cf8624367e6

* add commit to trigger gh actions

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <[email protected]>

* add setup/cleanup test resources scripts

* Update samples/snippets/remove_test_resources.py

Co-authored-by: Anthonios Partheniou <[email protected]>

* review comments fix

Co-authored-by: Anthonios Partheniou <[email protected]>
Co-authored-by: gcf-owl-bot[bot] <78513119+gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

samples/snippets/TEST_RESOURCES_SETUP_CLEANUP.md

@@ -0,0 +1,39 @@
+# How to set up/ tear down the test resources
+
+## Required environment variables
+
+To successfully import the catalog data for tests, the following environment variables should be set:
+ - PROJECT_NUMBER
+ - BUCKET_NAME
+These values are stored in the Secret Manager and will be submitted as 
+   docker environment variables before the test run.
+
+The Secret Manager name is set in .kokoro/presubmit/common.cfg file, SECRET_MANAGER_KEYS variable.
+
+## Import catalog data
+
+There is a JSON file with valid products prepared in the `product` directory:
+`resources/products.json`.
+
+Run the `create_test_resources.py` to perform the following actions:
+   - create the GCS bucket <BUCKET_NAME>, 
+   - upload the product data from `resources/products.json` file,
+   - import products to the default branch of the Retail catalog.
+
+```
+$ python create_test_resources.py
+```
+
+In the result 316 products should be created in the test project catalog.
+
+
+## Remove catalog data
+
+Run the `remove_test_resources.py` to perform the following actions:
+   - remove all objects from the GCS bucket <BUCKET_NAME>, 
+   - remove the <BUCKET_NAME> bucket,
+   - delete all products from the Retail catalog.
+
+```
+$ python remove_test_resources.py
+```

samples/snippets/create_test_resources.py

@@ -0,0 +1,127 @@
+# Copyright 2021 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import re
+import time
+
+from google.cloud.storage.bucket import Bucket
+
+from google.cloud import storage
+from google.cloud.retail import GcsSource, ImportErrorsConfig, \
+    ImportProductsRequest, ProductInputConfig
+from google.cloud.retail_v2 import ProductServiceClient
+
+project_number = os.getenv('PROJECT_NUMBER')
+bucket_name = os.getenv('BUCKET_NAME')
+storage_client = storage.Client()
+resource_file = "resources/products.json"
+object_name = re.search('resources/(.*?)$', resource_file).group(1)
+default_catalog = "projects/{0}/locations/global/catalogs/default_catalog/branches/default_branch".format(
+    project_number)
+
+
+def create_bucket(bucket_name: str) -> Bucket:
+    """Create a new bucket in Cloud Storage"""
+    print("Creating new bucket:" + bucket_name)
+    bucket_exists = check_if_bucket_exists(bucket_name)
+    if bucket_exists:
+        print("Bucket {} already exists".format(bucket_name))
+        return storage_client.bucket(bucket_name)
+    else:
+        bucket = storage_client.bucket(bucket_name)
+        bucket.storage_class = "STANDARD"
+        new_bucket = storage_client.create_bucket(bucket, location="us")
+        print(
+            "Created bucket {} in {} with storage class {}".format(
+                new_bucket.name, new_bucket.location, new_bucket.storage_class
+            )
+        )
+        return new_bucket
+
+
+def check_if_bucket_exists(new_bucket_name):
+    """Check if bucket is already exists"""
+    bucket_exists = False
+    buckets = storage_client.list_buckets()
+    for bucket in buckets:
+        if bucket.name == new_bucket_name:
+            bucket_exists = True
+            break
+    return bucket_exists
+
+
+def upload_data_to_bucket(bucket: Bucket):
+    """Upload data to a GCS bucket"""
+    blob = bucket.blob(object_name)
+    blob.upload_from_filename(resource_file)
+    print("Data from {} has being uploaded to {}".format(resource_file,
+                                                         bucket.name))
+
+
+def get_import_products_gcs_request():
+    """Get import products from gcs request"""
+    gcs_bucket = "gs://{}".format(bucket_name)
+    gcs_errors_bucket = "{}/error".format(gcs_bucket)
+
+    gcs_source = GcsSource()
+    gcs_source.input_uris = ["{0}/{1}".format(gcs_bucket, object_name)]
+
+    input_config = ProductInputConfig()
+    input_config.gcs_source = gcs_source
+
+    errors_config = ImportErrorsConfig()
+    errors_config.gcs_prefix = gcs_errors_bucket
+
+    import_request = ImportProductsRequest()
+    import_request.parent = default_catalog
+    import_request.reconciliation_mode = ImportProductsRequest.ReconciliationMode.INCREMENTAL
+    import_request.input_config = input_config
+    import_request.errors_config = errors_config
+
+    print("---import products from google cloud source request---")
+    print(import_request)
+
+    return import_request
+
+
+def import_products_from_gcs():
+    """Call the Retail API to import products"""
+    import_gcs_request = get_import_products_gcs_request()
+    gcs_operation = ProductServiceClient().import_products(
+        import_gcs_request)
+    print(
+        "Import operation is started: {}".format(gcs_operation.operation.name))
+
+    while not gcs_operation.done():
+        print("Please wait till operation is completed")
+        time.sleep(5)
+    print("Import products operation is completed")
+
+    if gcs_operation.metadata is not None:
+        print("Number of successfully imported products")
+        print(gcs_operation.metadata.success_count)
+        print("Number of failures during the importing")
+        print(gcs_operation.metadata.failure_count)
+    else:
+        print("Operation.metadata is empty")
+
+    print(
+        "Wait 2 -5 minutes till products become indexed in the catalog,\
+after that they will be available for search")
+
+
+created_bucket = create_bucket(bucket_name)
+upload_data_to_bucket(created_bucket)
+import_products_from_gcs()

samples/snippets/remove_test_resources.py

@@ -0,0 +1,73 @@
+# Copyright 2021 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+
+from google.api_core.exceptions import PermissionDenied
+from google.cloud.storage.bucket import Bucket
+
+from google.cloud import storage
+from google.cloud.retail import DeleteProductRequest, ListProductsRequest, \
+    ProductServiceClient
+
+project_number = os.getenv('PROJECT_NUMBER')
+bucket_name = os.getenv('BUCKET_NAME')
+
+default_catalog = "projects/{0}/locations/global/catalogs/default_catalog/branches/default_branch".format(
+    project_number)
+
+storage_client = storage.Client()
+
+
+def delete_bucket():
+    """Delete bucket"""
+    try:
+        bucket = storage_client.get_bucket(bucket_name)
+    except:
+        print("Bucket {} does not exists".format(bucket_name))
+    else:
+        delete_object_from_bucket(bucket)
+        bucket.delete()
+        print("bucket {} is deleted".format(bucket_name))
+
+
+def delete_object_from_bucket(bucket: Bucket):
+    """Delete object from bucket"""
+    blobs = bucket.list_blobs()
+    for blob in blobs:
+        blob.delete()
+    print("all objects are deleted from GCS bucket {}".format(bucket.name))
+
+
+def delete_all_products():
+    """Delete all products in the catalog"""
+    product_client = ProductServiceClient()
+    list_request = ListProductsRequest()
+    list_request.parent = default_catalog
+    products = product_client.list_products(list_request)
+    delete_count = 0
+    for product in products:
+        delete_request = DeleteProductRequest()
+        delete_request.name = product.name
+        try:
+            product_client.delete_product(delete_request)
+            delete_count += 1
+        except PermissionDenied:
+            print(
+                "Ignore PermissionDenied in case the product does not exist at time of deletion")
+    print(f"{delete_count} products were deleted from {default_catalog}")
+
+
+delete_bucket()
+delete_all_products()