Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Cloud Asset feeds #3750

Merged
merged 17 commits into from
Jul 21, 2020
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
252 changes: 252 additions & 0 deletions products/cloudasset/api.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,252 @@
# Copyright 2019 Google Inc.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind changing the copyright to 2020, please? This is applicable to the other files as well. Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
--- !ruby/object:Api::Product
name: CloudAsset
display_name: Cloud Asset Inventory
versions:
- !ruby/object:Api::Product::Version
name: ga
base_url: https://cloudasset.googleapis.com/v1/
scopes:
- https://www.googleapis.com/auth/cloud-platform
apis_required:
- !ruby/object:Api::Product::ApiReference
name: Cloud Asset API
url: https://console.cloud.google.com/apis/library/cloudasset.googleapis.com/
objects:
- !ruby/object:Api::Resource
name: ProjectFeed
base_url: projects/{{project}}/feeds
create_url: projects/{{project}}/feeds?feedId={{feed_id}}
self_link: "{{name}}"
update_verb: :PATCH
update_mask: true
collection_url_key: 'feeds'
description: |
Describes a Cloud Asset Inventory feed used to to listen to asset updates.
references: !ruby/object:Api::Resource::ReferenceLinks
guides:
'Official Documentation':
'https://cloud.google.com/asset-inventory/docs'
api: 'https://cloud.google.com/asset-inventory/docs/reference/rest/'
properties:
- !ruby/object:Api::Type::String
name: billing_project
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind explaining what billing_project is used for here? I think if you're looking to override the project, you should be able to just call it project. Is that the intent? It also would not need to be url_param_only.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See an explanation for this in this comment: #3750 (comment)

url_param_only: true
input: true
description: |
The project whose identity will be used when sending messages to the destination pubsub topic.
- !ruby/object:Api::Type::String
name: name
output: true
description: |
The format will be projects/{projectNumber}/feeds/{client-assigned_feed_identifier}.
- !ruby/object:Api::Type::String
name: feedId
description: |
This is the client-assigned asset feed identifier and it needs to be unique under a specific parent.
required: true
input: true
url_param_only: true
- !ruby/object:Api::Type::Array
megan07 marked this conversation as resolved.
Show resolved Hide resolved
name: assetNames
item_type: Api::Type::String
description: |
A list of the full names of the assets to receive updates. You must specify either or both of
assetNames and assetTypes. Only asset updates matching specified assetNames and assetTypes are
exported to the feed. For example: //compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1.
See https://cloud.google.com/apis/design/resourceNames#fullResourceName for more info.
- !ruby/object:Api::Type::Array
name: assetTypes
item_type: Api::Type::String
description: |
A list of types of the assets to receive updates. You must specify either or both of assetNames
and assetTypes. Only asset updates matching specified assetNames and assetTypes are exported to
the feed. For example: "compute.googleapis.com/Disk"
See https://cloud.google.com/asset-inventory/docs/supported-asset-types for a list of all
supported asset types.
- !ruby/object:Api::Type::String
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be Type::Enum and we can list the values here. This is applicable for all the different types of feeds.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah! Good to know. I did not know how to indicate that. The contentType is indeed an Enum. I'll change that.

name: contentType
description: |
Asset content type. If not specified, no content but the asset name and type will be returned.
- !ruby/object:Api::Type::NestedObject
name: feedOutputConfig
required: true
description: |
Output configuration for asset feed destination.
properties:
- !ruby/object:Api::Type::NestedObject
name: pubsubDestination
description: |
Destination on Cloud Pubsub.
properties:
- !ruby/object:Api::Type::String
name: topic
description: |
Destination on Cloud Pubsub topic.
- !ruby/object:Api::Resource
name: FolderFeed
base_url: "{{folder}}/feeds"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want the prefix folders in the base_url and create_url? I assume it's not here because the name returned from the google_folder resource/data source already has the prefix, but as I'm looking over it, I wonder if maybe we want to add a computed attribute, folder_id on the folder, that would be just the folder id rather than the full name. It should be a quick add. What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are right. I did not like it either. I'll try to change this, and come back if I have any question on how to do it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't like this either. I have added the folder_id computed attribute as suggested.

create_url: "{{folder}}/feeds?feedId={{feed_id}}"
self_link: "{{name}}"
update_verb: :PATCH
update_mask: true
collection_url_key: 'feeds'
description: |
Describes a Cloud Asset Inventory feed used to to listen to asset updates.
references: !ruby/object:Api::Resource::ReferenceLinks
guides:
'Official Documentation':
'https://cloud.google.com/asset-inventory/docs'
api: 'https://cloud.google.com/asset-inventory/docs/reference/rest/'
properties:
- !ruby/object:Api::Type::String
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might want to put this under a parameters block rather than properties

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the difference between the two but, if it seems right, Why not?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is there's not really a technical difference between the two (the technical importance is url_param_only), but more of an organizational difference. Thanks!

name: folder
required: true
input: true
url_param_only: true
description: |
The folder this feed should be created in.
- !ruby/object:Api::Type::String
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary for folder feeds? I think it should be handled with supports_indirect_user_project_override

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I need this. This is where this API is a bit different. When creating a feed, you need to provide the "billing project" that will be used for the feed in the X-Goog-User-Project header. But this project id will not appear in the resource URL.
I enabled the supports_indirect_user_project_override flag because I needed the project variable to be defined and passed to the sendRequestWithTimeout function. Otherwise a hard coded empty string is passed.
This is where I am not happy in the way we are passing this billing project today. The current implementation of the sendRequestWithTimeout function has been designed only for a configuration global billing project. But this API requires the flexibility to use different billing projects for different feeds.
The billing project affects the following:
1.- It is the project that will need to have the cloudasset.googleapis.com service enabled.
2.- It is the project whose cloudasset identity will be used for publishing messages to the destination topic.
So we need full flexibility here. And this is why in the projectFeed resource we have a "project" parameter and a "billing_project". If no billing_project is provided, the "project" will also be used as billing_project.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'm starting to understand this a bit better I think. Will the billing_project always be the project in feed_ouput_config.pubsub_destination.topic? I'm curious if we can make this a smoother, reusable solution for in the future. Something similar to supports_indirect_user_project_override, but a way to point to a resource's url that has the project we want. If that doesn't seem plausible, or the work seems too much, I'm not against what you have, it seems to work fine!

name: project
required: true
input: true
url_param_only: true
description: |
The project whose identity will be used when sending messages to the destination pubsub topic.
- !ruby/object:Api::Type::String
name: name
output: true
description: |
The format will be folders/{folder_number}/feeds/{client-assigned_feed_identifier}.
- !ruby/object:Api::Type::String
name: feedId
description: |
This is the client-assigned asset feed identifier and it needs to be unique under a specific parent.
required: true
input: true
url_param_only: true
- !ruby/object:Api::Type::Array
name: assetNames
item_type: Api::Type::String
description: |
A list of the full names of the assets to receive updates. You must specify either or both of
assetNames and assetTypes. Only asset updates matching specified assetNames and assetTypes are
exported to the feed. For example: //compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1.
See https://cloud.google.com/apis/design/resourceNames#fullResourceName for more info.
- !ruby/object:Api::Type::Array
name: assetTypes
item_type: Api::Type::String
description: |
A list of types of the assets to receive updates. You must specify either or both of assetNames
and assetTypes. Only asset updates matching specified assetNames and assetTypes are exported to
the feed. For example: "compute.googleapis.com/Disk"
See https://cloud.google.com/asset-inventory/docs/supported-asset-types for a list of all
supported asset types.
- !ruby/object:Api::Type::String
name: contentType
description: |
Asset content type. If not specified, no content but the asset name and type will be returned.
- !ruby/object:Api::Type::NestedObject
name: feedOutputConfig
required: true
description: |
Output configuration for asset feed destination.
properties:
- !ruby/object:Api::Type::NestedObject
name: pubsubDestination
description: |
Destination on Cloud Pubsub.
properties:
- !ruby/object:Api::Type::String
name: topic
description: |
Destination on Cloud Pubsub topic.
- !ruby/object:Api::Resource
name: OrganizationFeed
base_url: "organizations/{{org_id}}/feeds"
create_url: "organizations/{{org_id}}/feeds?feedId={{feed_id}}"
self_link: "{{name}}"
update_verb: :PATCH
update_mask: true
collection_url_key: 'feeds'
description: |
Describes a Cloud Asset Inventory feed used to to listen to asset updates.
references: !ruby/object:Api::Resource::ReferenceLinks
guides:
'Official Documentation':
'https://cloud.google.com/asset-inventory/docs'
api: 'https://cloud.google.com/asset-inventory/docs/reference/rest/'
properties:
- !ruby/object:Api::Type::String
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want to put this under a parameters block as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned earlier, seems right.

name: org_id
required: true
input: true
url_param_only: true
description: |
The organization this feed should be created in.
- !ruby/object:Api::Type::String
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary for organization feeds? I think this would be handled with supports_indirect_user_project_override

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Just like in the case of the FolderFeed, I am using this flag only to get the "project" var declared before the sendRequestWithTimeout function is called. Not the way I'd like to do it, but currently the only way to set the billing project header.

name: project
required: true
input: true
url_param_only: true
description: |
The project whose identity will be used when sending messages to the destination pubsub topic.
- !ruby/object:Api::Type::String
name: name
output: true
description: |
The format will be organizations/{organization_number}/feeds/{client-assigned_feed_identifier}.
- !ruby/object:Api::Type::String
name: feedId
description: |
This is the client-assigned asset feed identifier and it needs to be unique under a specific parent.
required: true
input: true
url_param_only: true
- !ruby/object:Api::Type::Array
name: assetNames
item_type: Api::Type::String
description: |
A list of the full names of the assets to receive updates. You must specify either or both of
assetNames and assetTypes. Only asset updates matching specified assetNames and assetTypes are
exported to the feed. For example: //compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1.
See https://cloud.google.com/apis/design/resourceNames#fullResourceName for more info.
- !ruby/object:Api::Type::Array
name: assetTypes
item_type: Api::Type::String
description: |
A list of types of the assets to receive updates. You must specify either or both of assetNames
and assetTypes. Only asset updates matching specified assetNames and assetTypes are exported to
the feed. For example: "compute.googleapis.com/Disk"
See https://cloud.google.com/asset-inventory/docs/supported-asset-types for a list of all
supported asset types.
- !ruby/object:Api::Type::String
name: contentType
description: |
Asset content type. If not specified, no content but the asset name and type will be returned.
- !ruby/object:Api::Type::NestedObject
name: feedOutputConfig
required: true
description: |
Output configuration for asset feed destination.
properties:
- !ruby/object:Api::Type::NestedObject
name: pubsubDestination
description: |
Destination on Cloud Pubsub.
properties:
- !ruby/object:Api::Type::String
name: topic
description: |
Destination on Cloud Pubsub topic.
65 changes: 65 additions & 0 deletions products/cloudasset/terraform.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Copyright 2019 Google Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

--- !ruby/object:Provider::Terraform::Config
overrides: !ruby/object:Overrides::ResourceOverrides
ProjectFeed: !ruby/object:Overrides::Terraform::ResourceOverride
autogen_async: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is necessary, I think it is the default

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right! I'm removing these.

custom_code: !ruby/object:Provider::Terraform::CustomCode
pre_create: templates/terraform/pre_create/cloud_asset_feed.go.erb
post_create: templates/terraform/post_create/cloud_asset_feed.go.erb
pre_update: templates/terraform/pre_update/cloud_asset_feed.go.erb
custom_import: templates/terraform/custom_import/cloud_asset_feed.go.erb
examples:
- !ruby/object:Provider::Terraform::Examples
name: "cloud_asset_project_feed"
primary_resource_id: "project_feed"
vars:
feed_id: "network-updates"
test_env_vars:
project: :PROJECT_NAME
FolderFeed: !ruby/object:Overrides::Terraform::ResourceOverride
autogen_async: false
supports_indirect_user_project_override: true
custom_code: !ruby/object:Provider::Terraform::CustomCode
pre_create: templates/terraform/pre_create/cloud_asset_feed.go.erb
post_create: templates/terraform/post_create/cloud_asset_feed.go.erb
pre_update: templates/terraform/pre_update/cloud_asset_feed.go.erb
custom_import: templates/terraform/custom_import/cloud_asset_feed.go.erb
examples:
- !ruby/object:Provider::Terraform::Examples
name: "cloud_asset_folder_feed"
primary_resource_id: "folder_feed"
vars:
feed_id: "network-updates"
folder_name: "Networking"
test_env_vars:
project: :PROJECT_NAME
org_id: :ORG_ID
OrganizationFeed: !ruby/object:Overrides::Terraform::ResourceOverride
autogen_async: false
supports_indirect_user_project_override: true
custom_code: !ruby/object:Provider::Terraform::CustomCode
pre_create: templates/terraform/pre_create/cloud_asset_feed.go.erb
post_create: templates/terraform/post_create/cloud_asset_feed.go.erb
pre_update: templates/terraform/pre_update/cloud_asset_feed.go.erb
custom_import: templates/terraform/custom_import/cloud_asset_feed.go.erb
examples:
- !ruby/object:Provider::Terraform::Examples
name: "cloud_asset_organization_feed"
primary_resource_id: "organization_feed"
vars:
feed_id: "network-updates"
test_env_vars:
project: :PROJECT_NAME
org_id: :ORG_ID
4 changes: 4 additions & 0 deletions provider/terraform/custom_code.rb
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,9 @@ class CustomCode < Api::Object
# (e.g. "fooBarDiffSuppress") and regexes that are necessarily
# exported (e.g. "fooBarValidationRegex").
attr_reader :constants
# This code is run before the Create call happens. It's placed
# in the Create function, just before the Create call is made.
attr_reader :pre_create
# This code is run after the Create call succeeds. It's placed
# in the Create function directly without modification.
attr_reader :post_create
Expand Down Expand Up @@ -126,6 +129,7 @@ def validate
check :update_encoder, type: String
check :decoder, type: String
check :constants, type: String
check :pre_create, type: String
check :post_create, type: String
check :custom_create, type: String
check :pre_update, type: String
Expand Down
4 changes: 4 additions & 0 deletions templates/terraform/custom_import/cloud_asset_feed.go.erb
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
if err := d.Set("name", d.Id()); err != nil {
return nil, err
}
return []*schema.ResourceData{d}, nil
51 changes: 51 additions & 0 deletions templates/terraform/examples/cloud_asset_folder_feed.tf.erb
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Create a feed that sends notifications about network resource updates under a
# particular folder.
resource "google_cloud_asset_folder_feed" "<%= ctx[:primary_resource_id] %>" {
project = "<%= ctx[:test_env_vars]["project"] %>"
folder = google_folder.my_folder.name
feed_id = "<%= ctx[:vars]["feed_id"] %>"
content_type = "RESOURCE"

asset_types = [
"compute.googleapis.com/Subnetwork",
"compute.googleapis.com/Network",
]

feed_output_config {
pubsub_destination {
topic = google_pubsub_topic.feed_output.id
}
}

# Wait for the permission to be ready on the destination topic.
depends_on = [
google_pubsub_topic_iam_member.cloud_asset_writer,
]
}

# The topic where the resource change notifications will be sent.
resource "google_pubsub_topic" "feed_output" {
project = "<%= ctx[:test_env_vars]["project"] %>"
name = "<%= ctx[:vars]["feed_id"] %>"
}

# The folder that will be monitored for resource updates.
resource "google_folder" "my_folder" {
display_name = "<%= ctx[:vars]["folder_name"] %>"
parent = "organizations/<%= ctx[:test_env_vars]["org_id"] %>"
}

# Find the project number of the project whose identity will be used for sending
# the asset change notifications.
data "google_project" "project" {
project_id = "<%= ctx[:test_env_vars]["project"] %>"
}

# Allow the publishing role to the Cloud Asset service account of the project that
# was used for sending the notifications.
resource "google_pubsub_topic_iam_member" "cloud_asset_writer" {
project = "<%= ctx[:test_env_vars]["project"] %>"
topic = google_pubsub_topic.feed_output.id
role = "roles/pubsub.publisher"
member = "serviceAccount:service-${data.google_project.project.number}@gcp-sa-cloudasset.iam.gserviceaccount.com"
}
Loading