Cloud Storage Rest API: Downloading objects

[hmac]

Hi! Thanks for this library. I've run into an issue related to the google cloud storage API.

Apparently the way to download the contents of an object via the cloud storage API is to include an alt=media parameter to the GET https://storage.googleapis.com/storage/v1/b/bucket/o/object endpoint. This corresponds to the storage_objects_get function:

pub async fn storage_objects_get(
    configuration: &configuration::Configuration,
    params: StoragePeriodObjectsPeriodGetParams,
) -> Result<
    crate::google_rest_apis::storage_v1::models::Object,
    Error<StoragePeriodObjectsPeriodGetError>>

Unfortunately this function expects the API to return an Object instance, so when you pass "media" as the alt parameter, the result is a serde deserialisation error.

I've fixed this locally by adding a new function storage_objects_get_bytes which always provides alt=media and has this signature:

pub async fn storage_objects_get_bytes(
    configuration: &configuration::Configuration,
    params: StoragePeriodObjectsPeriodGetParams,
) -> Result<
    bytes::Bytes,
    Error<StoragePeriodObjectsPeriodGetError>>

This is my local change, but I'm not sure if I've put it in the right place to play nice with the codegen.

diff --git a/gcloud-sdk/src/rest_apis/google_rest_apis/storage_v1/apis/objects_api.rs b/gcloud-sdk/src/rest_apis/google_rest_apis/storage_v1/apis/objects_api.rs
index 73133147b..73617b7be 100644
--- a/gcloud-sdk/src/rest_apis/google_rest_apis/storage_v1/apis/objects_api.rs
+++ b/gcloud-sdk/src/rest_apis/google_rest_apis/storage_v1/apis/objects_api.rs
@@ -1177,6 +1177,134 @@ pub async fn storage_objects_get(
     }
 }
 
+/// Retrieves the contents of an object
+pub async fn storage_objects_get_bytes(
+    configuration: &configuration::Configuration,
+    params: StoragePeriodObjectsPeriodGetParams,
+) -> Result<
+    bytes::Bytes,
+    Error<StoragePeriodObjectsPeriodGetError>,
+> {
+    let local_var_configuration = configuration;
+
+    // unbox the parameters
+    let bucket = params.bucket;
+    let object = params.object;
+    let fields = params.fields;
+    let key = params.key;
+    let oauth_token = params.oauth_token;
+    let pretty_print = params.pretty_print;
+    let quota_user = params.quota_user;
+    let upload_type = params.upload_type;
+    let user_ip = params.user_ip;
+    let generation = params.generation;
+    let if_generation_match = params.if_generation_match;
+    let if_generation_not_match = params.if_generation_not_match;
+    let if_metageneration_match = params.if_metageneration_match;
+    let if_metageneration_not_match = params.if_metageneration_not_match;
+    let projection = params.projection;
+    let user_project = params.user_project;
+
+    let local_var_client = &local_var_configuration.client;
+
+    let local_var_uri_str = format!(
+        "{}/b/{bucket}/o/{object}",
+        local_var_configuration.base_path,
+        bucket = crate::google_rest_apis::storage_v1::apis::urlencode(bucket),
+        object = crate::google_rest_apis::storage_v1::apis::urlencode(object)
+    );
+    let mut local_var_req_builder =
+        local_var_client.request(reqwest::Method::GET, local_var_uri_str.as_str());
+
+    // The user-supplied `alt` parameter is ignored, because it must be "media"
+    local_var_req_builder = local_var_req_builder.query(&[("alt", "media".to_string())]);
+
+    if let Some(ref local_var_str) = fields {
+        local_var_req_builder =
+            local_var_req_builder.query(&[("fields", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = key {
+        local_var_req_builder = local_var_req_builder.query(&[("key", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = oauth_token {
+        local_var_req_builder =
+            local_var_req_builder.query(&[("oauth_token", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = pretty_print {
+        local_var_req_builder =
+            local_var_req_builder.query(&[("prettyPrint", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = quota_user {
+        local_var_req_builder =
+            local_var_req_builder.query(&[("quotaUser", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = upload_type {
+        local_var_req_builder =
+            local_var_req_builder.query(&[("uploadType", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = user_ip {
+        local_var_req_builder =
+            local_var_req_builder.query(&[("userIp", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = generation {
+        local_var_req_builder =
+            local_var_req_builder.query(&[("generation", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = if_generation_match {
+        local_var_req_builder =
+            local_var_req_builder.query(&[("ifGenerationMatch", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = if_generation_not_match {
+        local_var_req_builder =
+            local_var_req_builder.query(&[("ifGenerationNotMatch", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = if_metageneration_match {
+        local_var_req_builder =
+            local_var_req_builder.query(&[("ifMetagenerationMatch", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = if_metageneration_not_match {
+        local_var_req_builder = local_var_req_builder
+            .query(&[("ifMetagenerationNotMatch", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = projection {
+        local_var_req_builder =
+            local_var_req_builder.query(&[("projection", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_str) = user_project {
+        local_var_req_builder =
+            local_var_req_builder.query(&[("userProject", &local_var_str.to_string())]);
+    }
+    if let Some(ref local_var_user_agent) = local_var_configuration.user_agent {
+        local_var_req_builder =
+            local_var_req_builder.header(reqwest::header::USER_AGENT, local_var_user_agent.clone());
+    }
+    if let Some(ref local_var_token) = local_var_configuration.oauth_access_token {
+        local_var_req_builder = local_var_req_builder.bearer_auth(local_var_token.to_owned());
+    };
+    if let Some(ref local_var_token) = local_var_configuration.oauth_access_token {
+        local_var_req_builder = local_var_req_builder.bearer_auth(local_var_token.to_owned());
+    };
+
+    let local_var_req = local_var_req_builder.build()?;
+    let local_var_resp = local_var_client.execute(local_var_req).await?;
+
+    let local_var_status = local_var_resp.status();
+
+    if !local_var_status.is_client_error() && !local_var_status.is_server_error() {
+        Ok(local_var_resp.bytes().await?)
+    } else {
+        let local_var_content = local_var_resp.text().await?;
+        let local_var_entity: Option<StoragePeriodObjectsPeriodGetError> =
+            serde_json::from_str(&local_var_content).ok();
+        let local_var_error = ResponseContent {
+            status: local_var_status,
+            content: local_var_content,
+            entity: local_var_entity,
+        };
+        Err(Error::ResponseError(local_var_error))
+    }
+}
+
 /// Returns an IAM policy for the specified object.
 pub async fn storage_objects_get_iam_policy(
     configuration: &configuration::Configuration,

What do you suggest is the best approach here?

[abdolence]

Thanks for reporting this 👍🏻

Oh dear, Google cloud storage API is a mess.
I already done this patch to the Open API spec here:

gcloud-protos-generator/openapi/google/storage-v1/ext/apis/objects_api.rs

So, I'd suggest to include some additional implementation as you did. The only thing we need is also the streaming version as well, so if you have big files not to allocate everything in the memory (for upload we have 2 version of the function for this).

Want to give a try with the PR, or I can look into this a bit later myself?

[abdolence]

Ok, I extracted your code to some extension functions here:
#53

Will release shortly! Thanks for sharing the code - it made my life a bit easier :)

[abdolence]

This is released now in
https://github.com/abdolence/gcloud-sdk-rs/releases/tag/v0.19.14

Feel free to reopen this if you find issues

[hmac]

Sorry for the late response, and thank you! That works perfectly :)

[abdolence]

Thanks for checking it!