Add ability to upload from local directory for cloud functions

[kardiff18]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

Cloud Functions lets you deploy via local directory, but currently terraform only supports uploads from GCS or Cloud Source Repos

New or Affected Resource(s)

• google_cloudfunctions_function

References

b/274818624

[paddycarver]

I'm not seeing any functionality for direct upload in the API. The closest I see is https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions/generateUploadUrl, which just uploads to cloud storage and then you use that URL as the upload URL when creating the function.

Can you link me to documentation for what you're describing, or can you describe what this would enable that you cannot currently do?

[kardiff18]

I currently do it via this quickstart <https://cloud.google.com/functions/docs/quickstart#deploy_a_function>, you just have to do it within the directory. It then supports local upload (and will let you upload multiple files at once.)

…

On Tue, Feb 19, 2019 at 6:20 PM Paddy ***@***.***> wrote: I'm not seeing any functionality for direct upload in the API. The closest I see is https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions/generateUploadUrl, which just uploads to cloud storage and then you use that URL as the upload URL when creating the function. Can you link me to documentation for what you're describing, or can you describe what this would enable that you cannot currently do? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3085 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVfYL7meEqHCceBTst73RYO5JcvGHSObks5vPIa5gaJpZM4bD9ie> .

-- *Kaitlin Ardiff* [email protected] Strategic Cloud Engineer

[paddycarver]

Can you run the relevant gcloud command and add --log-http with it, and then share the output that's written to stderr here?

[jessefoster]

@paddycarver
The gcloud cli does use generateUploadUrl API underneath. I included the http log, minus some secrets and polling.

gcloud beta functions deploy BatchRecord --log-http --trigger-http --memory 128MB --runtime go111 --region us-east1 --env-vars-file=.env.yaml --timeout=540
=======================
==== request start ====
uri: https://www.googleapis.com/oauth2/v4/token
method: POST
== headers start ==
content-type: application/x-www-form-urlencoded
user-agent: google-cloud-sdk gcloud/236.0.0 command/gcloud.beta.functions.deploy invocation-id/e7fe42280b5040b0a1cbe1f912008cf6 environment/None environment-version/None interactive/True from-script/False python/2.7.10 term/xterm-256color (Macintosh; Intel Mac OS X 17.7.0)
== headers end ==
== body start ==
Body redacted: Contains oauth token. Set log_http_redact_token property to false to print the body of this request.
== body end ==
==== request end ====
---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
cache-control: private
content-length: 1330
content-type: application/json; charset=utf-8
date: Wed, 20 Mar 2019 21:25:53 GMT
server: ESF
status: 200
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
-- headers end --
-- body start --
Body redacted: Contains oauth token. Set log_http_redact_token property to false to print the body of this response.
-- body end --
total round trip time (request+response): 0.198 secs
---- response end ----
----------------------
=======================

==== request start ====
uri: https://cloudfunctions.googleapis.com/v1/projects/.../locations/us-east1/functions/BatchRecord?alt=json
method: GET
== headers start ==
Authorization: --- Token Redacted ---
accept: application/json
accept-encoding: gzip, deflate
content-length: 0
user-agent: google-cloud-sdk x_Tw5K8nnjoRAqULM9PFAC2b gcloud/236.0.0 command/gcloud.beta.functions.deploy invocation-id/3c497f20696643178a7645b965165dba environment/None environment-version/None interactive/True from-script/False python/2.7.10 term/xterm-256color (Macintosh; Intel Mac OS X 17.7.0)
== headers end ==
== body start ==
== body end ==
==== request end ====

---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
cache-control: private
content-length: 1836
content-location: https://cloudfunctions.googleapis.com/v1/projects/.../locations/us-east1/functions/BatchRecord?alt=json
content-type: application/json; charset=UTF-8
date: Wed, 20 Mar 2019 21:25:54 GMT
server: ESF
status: 200
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
-- headers end --
-- body start --
{
  "name": "projects/.../locations/us-east1/functions/BatchRecord",
  "httpsTrigger": {
    "url": "https://us-east1-....cloudfunctions.net/BatchRecord"
  },
  "status": "ACTIVE",
  "entryPoint": "BatchRecord",
  "timeout": "540s",
  "availableMemoryMb": 128,
  "updateTime": "2019-03-06T20:47:51Z",
  "versionId": "1",
  "labels": {
    "deployment-tool": "cli-gcloud"
  },
  "sourceUploadUrl": "https://storage.googleapis.com/gcf-upload-us-east1-3141732f-f672-4768-81db-16c9bab9ab51/6308bfa8-153c-438b-8953-e0816e1ea423.zip?GoogleAccessId=service-452490176416@gcf-admin-robot.iam.gserviceaccount.com&Expires=1551906830&Signature=B%2FCGQ2VyGKKrTQ%2BdY%2BnRwW8OO%2FFPL6byy44%2FuBN5QfYsFk8u1k8ezUwJA9FcHlLgFJINzlGcsjLNF7zKNpviWJnWaVlcQ%2BV0WwxyDPD3jkzMeCYnNzObe%2FOzWIMmw9s2lfmUi6Z1%2Fadj1qHmS8%2FSo6jv3VOESLP%2BXvG%2FPWaUjR%2Fh1uNHhGvPaplg7EyXiuIDRi61nNNcsW3Vdj88XZYnifBIVjBz9vIidb52r2Y3RUBWk%2FgzWXWVP11E7Mn%2BHS5D2gSrkXCr8SnRzUhVXEMhj2rDWChA0TnC4VIUPSvW5aoQ7AEdHKhv%2FH8g8a1cUuiFhd4XuBcFb6sYGcXsNuqxfQ%3D%3D",
  "environmentVariables": {
    ...
  },
  "runtime": "go111"
}

-- body end --
total round trip time (request+response): 0.487 secs
---- response end ----
----------------------
=======================

==== request start ====
uri: https://cloudfunctions.googleapis.com/v1/projects/.../locations/us-east1/functions:generateUploadUrl?alt=json
method: POST
== headers start ==
Authorization: --- Token Redacted ---
accept: application/json
accept-encoding: gzip, deflate
content-length: 2
content-type: application/json
user-agent: google-cloud-sdk x_Tw5K8nnjoRAqULM9PFAC2b gcloud/236.0.0 command/gcloud.beta.functions.deploy invocation-id/8a79e2c03c9f4b398fda1418102c0e06 environment/None environment-version/None interactive/True from-script/False python/2.7.10 term/xterm-256color (Macintosh; Intel Mac OS X 17.7.0)
== headers end ==
== body start ==
{}
== body end ==
==== request end ====
---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
cache-control: private
content-length: 626
content-type: application/json; charset=UTF-8
date: Wed, 20 Mar 2019 21:25:54 GMT
server: ESF
status: 200
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
-- headers end --
-- body start --
{
  "uploadUrl": "https://storage.googleapis.com/gcf-upload-us-east1-3141732f-f672-4768-81db-16c9bab9ab51/fe8f3095-39f8-405d-af51-afb1f36cff2c.zip?GoogleAccessId=service-452490176416@gcf-admin-robot.iam.gserviceaccount.com&Expires=1553118954&Signature=Slj9Ci2C1IMEntxXK44uQNRegAwVsdbYvG0e1ptLsgqNxBEfgr9gH60YZcTxv8YrUkldXa2nzLeKPVd8tKpmG6KmArhCXPKT3upoezvOAyQs2ENbsOwA2r3VhUEaEaMsREv8oR1urNvKjna8RNo%2Bt%2FPNgWw24QM%2BADaliU1jVawele9xH4YS2%2Bpf%2BJL4ObaIW4NOgWpc2Hgl4jJRa6YQJXO2b%2Flz05S1UhTbd6grm%2BqJU3wdNjv6JHVILXhfjca94MJz7SXgql36YSkFej%2BdPuF8%2FdQKkBFjAgmkJTuy3C1QoDE1lH5v09e%2BGBxXWTgT%2F5lxDQhhzqVYf0oapCNb5Q%3D%3D"
}

-- body end --
total round trip time (request+response): 0.534 secs
---- response end ----
----------------------

==== request start ====
uri: https://storage.googleapis.com/gcf-upload-us-east1-3141732f-f672-4768-81db-16c9bab9ab51/fe8f3095-39f8-405d-af51-afb1f36cff2c.zip?GoogleAccessId=service-452490176416@gcf-admin-robot.iam.gserviceaccount.com&Expires=1553118954&Signature=Slj9Ci2C1IMEntxXK44uQNRegAwVsdbYvG0e1ptLsgqNxBEfgr9gH60YZcTxv8YrUkldXa2nzLeKPVd8tKpmG6KmArhCXPKT3upoezvOAyQs2ENbsOwA2r3VhUEaEaMsREv8oR1urNvKjna8RNo%2Bt%2FPNgWw24QM%2BADaliU1jVawele9xH4YS2%2Bpf%2BJL4ObaIW4NOgWpc2Hgl4jJRa6YQJXO2b%2Flz05S1UhTbd6grm%2BqJU3wdNjv6JHVILXhfjca94MJz7SXgql36YSkFej%2BdPuF8%2FdQKkBFjAgmkJTuy3C1QoDE1lH5v09e%2BGBxXWTgT%2F5lxDQhhzqVYf0oapCNb5Q%3D%3D
method: PUT
== headers start ==
Content-Length: 191836
content-length: 191836
content-type: application/zip
user-agent: gcloud/236.0.0 command/gcloud.beta.functions.deploy invocation-id/8818819fd74f4b86b793b4ab079573ec environment/None environment-version/None interactive/True from-script/False python/2.7.10 term/xterm-256color (Macintosh; Intel Mac OS X 17.7.0)
x-goog-content-length-range: 0,104857600
== headers end ==
== body start ==
== body end ==
==== request end ====
---- response start ----
-- headers start --
alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
content-length: 0
content-type: text/html; charset=UTF-8
date: Wed, 20 Mar 2019 21:25:55 GMT
etag: "3b85385d5de7a3534bf164465ff2e58a"
server: UploadServer
status: 200
vary: Origin
x-goog-generation: 1553117155199029
x-goog-hash: crc32c=gRZIaQ==, md5=O4U4XV3no1NL8WRGX/Llig==
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 191836
x-guploader-uploadid: AEnB2UrgLdMA0KdEgMEdFPK1ax9D5IsbMdVVzQ4FDq6DROEhO25l263wLH3oOFlThUEAIrJoMXEEyEVuJTWArRzN6d3aPVnmdw
-- headers end --
-- body start --
....
-- body end --
total round trip time (request+response): 0.343 secs
---- response end ----
----------------------
=======================

==== request start ====
uri: https://cloudfunctions.googleapis.com/v1/projects/.../locations/us-east1/functions/BatchRecord?alt=json&updateMask=availableMemoryMb%2CeventTrigger%2ChttpsTrigger%2Cruntime%2CsourceUploadUrl%2Ctimeout
method: PATCH
== headers start ==
Authorization: --- Token Redacted ---
accept: application/json
accept-encoding: gzip, deflate
content-length: 1669
content-type: application/json
user-agent: google-cloud-sdk x_Tw5K8nnjoRAqULM9PFAC2b gcloud/236.0.0 command/gcloud.beta.functions.deploy invocation-id/cb2831c92bd44bdfb1db8fa7e2b45f57 environment/None environment-version/None interactive/True from-script/False python/2.7.10 term/xterm-256color (Macintosh; Intel Mac OS X 17.7.0)
== headers end ==
== body start ==
{"availableMemoryMb": 128, "entryPoint": "BatchRecord", "environmentVariables": {}, "httpsTrigger": {}, "labels": {"deployment-tool": "cli-gcloud"}, "name": "projects/.../locations/us-east1/functions/BatchRecord", "runtime": "go111", "serviceAccountEmail": "", "sourceUploadUrl": "https://storage.googleapis.com/gcf-upload-us-east1-3141732f-f672-4768-81db-16c9bab9ab51/fe8f3095-39f8-405d-af51-afb1f36cff2c.zip?GoogleAccessId=service-452490176416@gcf-admin-robot.iam.gserviceaccount.com&Expires=1553118954&Signature=Slj9Ci2C1IMEntxXK44uQNRegAwVsdbYvG0e1ptLsgqNxBEfgr9gH60YZcTxv8YrUkldXa2nzLeKPVd8tKpmG6KmArhCXPKT3upoezvOAyQs2ENbsOwA2r3VhUEaEaMsREv8oR1urNvKjna8RNo%2Bt%2FPNgWw24QM%2BADaliU1jVawele9xH4YS2%2Bpf%2BJL4ObaIW4NOgWpc2Hgl4jJRa6YQJXO2b%2Flz05S1UhTbd6grm%2BqJU3wdNjv6JHVILXhfjca94MJz7SXgql36YSkFej%2BdPuF8%2FdQKkBFjAgmkJTuy3C1QoDE1lH5v09e%2BGBxXWTgT%2F5lxDQhhzqVYf0oapCNb5Q%3D%3D", "status": "ACTIVE", "timeout": "540s", "updateTime": "2019-03-06T20:47:51Z", "versionId": "1"}
== body end ==
==== request end ====
---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
cache-control: private
content-length: 2324
content-type: application/json; charset=UTF-8
date: Wed, 20 Mar 2019 21:25:55 GMT
server: ESF
status: 200
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
-- headers end --
-- body start --
{
  "name": "operations/ZG9tYWluLW5hbWUtc3lzdGVtLTIyMTcyMS91cy1lYXN0MS9CYXRjaFJlY29yZC9LcUdqdlM0RmU5SQ",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.functions.v1.OperationMetadataV1",
    "target": "projects/.../locations/us-east1/functions/BatchRecord",
    "type": "UPDATE_FUNCTION",
    "request": {
      "@type": "type.googleapis.com/google.cloud.functions.v1.CloudFunction",
      "name": "projects/.../locations/us-east1/functions/BatchRecord",
      "httpsTrigger": {},
      "status": "ACTIVE",
      "entryPoint": "BatchRecord",
      "timeout": "540s",
      "availableMemoryMb": 128,
      "updateTime": "2019-03-06T20:47:51Z",
      "versionId": "1",
      "labels": {
        "deployment-tool": "cli-gcloud"
      },
      "sourceUploadUrl": "https://storage.googleapis.com/gcf-upload-us-east1-3141732f-f672-4768-81db-16c9bab9ab51/fe8f3095-39f8-405d-af51-afb1f36cff2c.zip?GoogleAccessId=service-452490176416@gcf-admin-robot.iam.gserviceaccount.com&Expires=1553118954&Signature=Slj9Ci2C1IMEntxXK44uQNRegAwVsdbYvG0e1ptLsgqNxBEfgr9gH60YZcTxv8YrUkldXa2nzLeKPVd8tKpmG6KmArhCXPKT3upoezvOAyQs2ENbsOwA2r3VhUEaEaMsREv8oR1urNvKjna8RNo%2Bt%2FPNgWw24QM%2BADaliU1jVawele9xH4YS2%2Bpf%2BJL4ObaIW4NOgWpc2Hgl4jJRa6YQJXO2b%2Flz05S1UhTbd6grm%2BqJU3wdNjv6JHVILXhfjca94MJz7SXgql36YSkFej%2BdPuF8%2FdQKkBFjAgmkJTuy3C1QoDE1lH5v09e%2BGBxXWTgT%2F5lxDQhhzqVYf0oapCNb5Q%3D%3D",
      "environmentVariables": {
        ...
      },
      "runtime": "go111"
    },
    "versionId": "2",
    "updateTime": "2019-03-20T21:25:55Z"
  }
}

-- body end --
total round trip time (request+response): 0.676 secs
---- response end ----

[michaelbannister]

I've been using Terraform's archive_file resource to achieve this but I'm now having trouble because GCF's python38 build process runs Python's compileall module, which has this bug/limitation https://bugs.python.org/issue34990 which is triggered by the 2049 modification date that archive_file chooses for the files in the zip.

hashicorp/terraform-provider-archive#40 is a related issue on the archive_file provider; a change there could also provide a solution.

[jdiazdiv]

Any updates about this issue? I may try to fix it.

[rileykarson]

No updates- as highlighted above, it's possible to perform this workflow already using archive_file and the google_storage_bucket_object resource. The documentation examples showcase this using a preexisting zip, but interpolating an archive_file based one should be simple.

Closing this issue out would mean an improvement to be able to do so entirely in band within the resource, which is not currently planned work for the provider development team.

[melinath]

Note from triage: Moving this out of the backlog since it could be reasonable to implement if the product team determines it is a priority.