feat: add ability to upload entire directory, add samples for upload … #2118
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| /** | ||
| * Copyright 2022 Google LLC | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| * @experimental | ||
|
There was a problem hiding this comment. is this where this tag belongs? There was a problem hiding this comment. Looking at https://sapui5.hana.ondemand.com/sdk/docs/topics/eeaa5de14e5f4fc1ac796bc0c1ada5fb.html it only notes |
||
| */ | ||
|
|
||
| // sample-metadata: | ||
|
There was a problem hiding this comment. does this comment belong here? it's outside the region tags and im not entirely sure the purpose. do we have a similar comment in other samples? (i only checked 2 and I didnt see it) There was a problem hiding this comment. I noticed this in some of the newer samples that had been written, for example createBucketWithDualRegion. |
||
| // title: Download Folder With Transfer Manager | ||
| // description: Downloads a folder in parallel utilizing transfer manager. | ||
| // usage: node downloadFolderWithTransferManager.js <BUCKET_NAME> <FOLDER_NAME> | ||
|
|
||
| function main(bucketName = 'my-bucket', folderName = 'my-folder') { | ||
| // [START storage_download_folder_transfer_manager] | ||
| /** | ||
| * TODO(developer): Uncomment the following lines before running the sample. | ||
| */ | ||
| // The ID of your GCS bucket | ||
| // const bucketName = 'your-unique-bucket-name'; | ||
|
|
||
| // The ID of the GCS folder to download. The folder will be downloaded to the local path of the executing code. | ||
| // const folderName = 'your-folder-name'; | ||
|
There was a problem hiding this comment. does this support both relative and absolute? |
||
|
|
||
| // Imports the Google Cloud client library | ||
| const {Storage, TransferManager} = require('@google-cloud/storage'); | ||
|
|
||
| // Creates a client | ||
| const storage = new Storage(); | ||
|
|
||
| // Creates a transfer manager client | ||
| const transferManager = new TransferManager(storage.bucket(bucketName)); | ||
|
|
||
| async function downloadFolderWithTransferManager() { | ||
| // Downloads the folder | ||
| await transferManager.downloadManyFiles(folderName); | ||
|
|
||
| console.log( | ||
| `gs://${bucketName}/${folderName} downloaded to ${folderName}.` | ||
| ); | ||
| } | ||
|
|
||
| downloadFolderWithTransferManager().catch(console.error); | ||
| // [END storage_download_folder_transfer_manager] | ||
| } | ||
|
|
||
| process.on('unhandledRejection', err => { | ||
| console.error(err.message); | ||
| process.exitCode = 1; | ||
| }); | ||
| main(...process.argv.slice(2)); | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| /** | ||
| * Copyright 2022 Google LLC | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| * @experimental | ||
| */ | ||
|
|
||
| // sample-metadata: | ||
|
Comment on lines
+15
to
+18
|
||
| // title: Upload Directory With Transfer Manager | ||
| // description: Uploads a directory in parallel utilizing transfer manager. | ||
| // usage: node uploadFolderWithTransferManager.js <BUCKET_NAME> <DIRECTORY_NAME> | ||
|
|
||
| function main(bucketName = 'my-bucket', directoryName = 'my-directory') { | ||
| // [START storage_upload_directory_transfer_manager] | ||
| /** | ||
| * TODO(developer): Uncomment the following lines before running the sample. | ||
| */ | ||
| // The ID of your GCS bucket | ||
| // const bucketName = 'your-unique-bucket-name'; | ||
|
|
||
| // The local directory to upload | ||
| // const directoryName = 'your-directory'; | ||
|
|
||
| // Imports the Google Cloud client library | ||
| const {Storage, TransferManager} = require('@google-cloud/storage'); | ||
|
|
||
| // Creates a client | ||
| const storage = new Storage(); | ||
|
|
||
| // Creates a transfer manager client | ||
| const transferManager = new TransferManager(storage.bucket(bucketName)); | ||
|
There was a problem hiding this comment. What are your thoughts on accepting a string in TransferManager's constructor and creating a bucket from it? There was a problem hiding this comment. I like this idea. Let me tackle it in a separate PR as to not muddy the waters on this one. |
||
|
|
||
| async function uploadDirectoryWithTransferManager() { | ||
| // Uploads the directory | ||
| await transferManager.uploadManyFiles(directoryName); | ||
|
|
||
| console.log(`${directoryName} uploaded to ${bucketName}.`); | ||
| } | ||
|
|
||
| uploadDirectoryWithTransferManager().catch(console.error); | ||
| // [END storage_upload_directory_transfer_manager] | ||
| } | ||
|
|
||
| process.on('unhandledRejection', err => { | ||
| console.error(err.message); | ||
| process.exitCode = 1; | ||
| }); | ||
| main(...process.argv.slice(2)); | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sort of a nit, but both
directoryandfolderare used interchangeably throughout the PR. Given the updated param name usesdirectory, should we stick with that verbiage?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I used folder when downloading as GCS does not call them directories because technically they aren't. I used directory for uploading to represent a local disk directory. Do you think it is better to just stick with directory throughout? My concern was this may cause future confusion with upcoming features.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That distinction makes sense. The only thing that needs to be updated is the readmes:
Upload Folder With Transfer Manager