Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Dual Region Support #1814

Merged
merged 26 commits into from
Apr 6, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
b401be7
feat:
d-goog Mar 14, 2022
86eeaa7
feat: Custom Dual Region Support
d-goog Mar 14, 2022
e9cce46
Merge branch 'custom-dual-region' of https://github.com/googleapis/no…
d-goog Mar 14, 2022
1c084fb
test: Custom Dual Region Tests with logging
d-goog Mar 15, 2022
135e889
feat: Add Custom Dual Region sample
d-goog Mar 15, 2022
52f78f8
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Mar 15, 2022
d5226fb
chore: typo
d-goog Mar 16, 2022
131bdb8
test: Add tests for `location` as an Array
d-goog Mar 21, 2022
030ad25
refactor: Rename - drop 'Custom' from 'Dual Regions'
d-goog Mar 22, 2022
50fdf88
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Mar 22, 2022
08f4985
style: Drop 'Custom' prefix
d-goog Mar 22, 2022
f7dee29
revert: Remove tuple support
d-goog Mar 22, 2022
0df8bf5
refactor: Rename variables from `locationX` to `regionX`
d-goog Mar 22, 2022
406fb22
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Mar 22, 2022
b01162e
test: Update metadata assertions
d-goog Mar 23, 2022
c61c60d
Merge branch 'custom-dual-region' of https://github.com/googleapis/no…
d-goog Mar 23, 2022
3e5ee01
test: Update tests to avoid name collision
d-goog Mar 25, 2022
e8a455e
Merge branch 'main' of https://github.com/googleapis/nodejs-storage i…
d-goog Mar 25, 2022
4fe95af
test: debug test
d-goog Mar 25, 2022
b479e80
test: fix test
d-goog Mar 25, 2022
599d8fd
Merge branch 'main' of https://github.com/googleapis/nodejs-storage i…
d-goog Mar 25, 2022
3fa6ebd
chore: test immediately deleting a specified dual-region bucket
d-goog Mar 25, 2022
99e8646
chore: undo delete debug
d-goog Mar 25, 2022
bc16c40
docs: 'dual region' -> 'dual-region'
d-goog Apr 1, 2022
ae8d221
Merge branch 'main' of https://github.com/googleapis/nodejs-storage i…
d-goog Apr 4, 2022
530c407
docs: Fix `bucket-locations` -> `locations` link
d-goog Apr 5, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ Samples are in the [`samples/`](https://github.com/googleapis/nodejs-storage/tre
| Configure Retries | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/configureRetries.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/configureRetries.js,samples/README.md) |
| Copy File | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/copyFile.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/copyFile.js,samples/README.md) |
| Copy Old Version Of File. | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/copyOldVersionOfFile.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/copyOldVersionOfFile.js,samples/README.md) |
| Create a Dual-Region Bucket | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/createBucketWithDualRegion.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/createBucketWithDualRegion.js,samples/README.md) |
| Create Bucket With Storage Class and Location. | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/createBucketWithStorageClassAndLocation.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/createBucketWithStorageClassAndLocation.js,samples/README.md) |
| Create Bucket With Turbo Replication | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/createBucketWithTurboReplication.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/createBucketWithTurboReplication.js,samples/README.md) |
| Create New Bucket | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/createNewBucket.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/createNewBucket.js,samples/README.md) |
Expand Down
20 changes: 20 additions & 0 deletions samples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ objects to users via direct download.
* [Configure Retries](#configure-retries)
* [Copy File](#copy-file)
* [Copy Old Version Of File.](#copy-old-version-of-file.)
* [Create a Dual-Region Bucket](#create-a-dual-region-bucket)
* [Create Bucket With Storage Class and Location.](#create-bucket-with-storage-class-and-location.)
* [Create Bucket With Turbo Replication](#create-bucket-with-turbo-replication)
* [Create New Bucket](#create-new-bucket)
Expand Down Expand Up @@ -413,6 +414,25 @@ __Usage:__



### Create a Dual-Region Bucket

Create a Dual-Region Bucket with provided locations.

View the [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/createBucketWithDualRegion.js).

[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/createBucketWithDualRegion.js,samples/README.md)

__Usage:__


`node createBucketWithDualRegion.js <BUCKET_NAME> <REGION1> <REGION2>`


-----




### Create Bucket With Storage Class and Location.

Create Bucket With Storage Class and Location.
Expand Down
67 changes: 67 additions & 0 deletions samples/createBucketWithDualRegion.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

'use strict';

// sample-metadata:
// title: Create a Dual-Region Bucket
// description: Create a Dual-Region Bucket with provided locations.
// usage: node createBucketWithDualRegion.js <BUCKET_NAME> <REGION1> <REGION2>

function main(
bucketName = 'my-bucket',
region1 = 'US-EAST1',
region2 = 'US-WEST1'
) {
// [START storage_create_bucket_dual_region]
/**
* TODO(developer): Uncomment the following lines before running the sample.
*/
// The ID of your GCS bucket
// const bucketName = 'your-unique-bucket-name';

// The bucket's pair of regions. Case-insensitive.
// See this documentation for other valid locations:
// https://cloud.google.com/storage/docs/locations
// const region1 = 'US-EAST1';
// const region2 = 'US-WEST1';

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');

// Creates a client
// The bucket in the sample below will be created in the project associated with this client.
// For more information, please see https://cloud.google.com/docs/authentication/production or https://googleapis.dev/nodejs/storage/latest/Storage.html
const storage = new Storage();

async function createDualRegionBucket() {
// For regions supporting dual-regions see: https://cloud.google.com/storage/docs/locations
const [bucket] = await storage.createBucket(bucketName, {
location: `${region1}+${region2}`, // e.g. `US-EAST1+US-WEST1`
});

console.log(`${bucket.name} created in '${region1}+${region2}'`);
}

createDualRegionBucket().catch(console.error);
// [END storage_create_bucket_dual_region]
}

process.on('unhandledRejection', err => {
console.error(err.message);
process.exitCode = 1;
});
main(...process.argv.slice(2));
2 changes: 1 addition & 1 deletion samples/createBucketWithStorageClassAndLocation.js
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ function main(

// The name of a location
// See this documentation for other valid locations:
// http://g.co/cloud/storage/docs/bucket-locations#location-mr
// http://g.co/cloud/storage/docs/locations#location-mr
// const location = 'ASIA';

// Imports the Google Cloud client library
Expand Down
50 changes: 36 additions & 14 deletions samples/system-test/buckets.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,17 +26,20 @@ const storage = new Storage();
const samplesTestBucketPrefix = `nodejs-storage-samples-${uuid.v4()}`;
const bucketName = `${samplesTestBucketPrefix}-a`;
const bucketNameDualRegion = `${samplesTestBucketPrefix}-b`;
const bucketNameWithClassAndLocation = `${samplesTestBucketPrefix}-c`;
const bucketNameDualRegionTurbo = `${samplesTestBucketPrefix}-c`;
const bucketNameWithClassAndLocation = `${samplesTestBucketPrefix}-d`;
const defaultKmsKeyName = process.env.GOOGLE_CLOUD_KMS_KEY_ASIA;
const bucket = storage.bucket(bucketName);
const bucketWithClassAndLocation = storage.bucket(
bucketNameWithClassAndLocation
);
const dualRegionBucket = storage.bucket(bucketNameDualRegion);
const dualRegionBucketTurbo = storage.bucket(bucketNameDualRegionTurbo);

const PUBLIC_ACCESS_PREVENTION_INHERITED = 'inherited';
const PUBLIC_ACCESS_PREVENTION_ENFORCED = 'enforced';

const DUAL_REGION = ['US-EAST1', 'US-WEST1'];
const RPO_ASYNC_TURBO = 'ASYNC_TURBO';
const RPO_DEFAULT = 'DEFAULT';

Expand Down Expand Up @@ -212,54 +215,73 @@ it('should set public access prevention to inherited', async () => {
);
});

it('should create a dual-region bucket', async () => {
const dualRegion = `${DUAL_REGION[0]}+${DUAL_REGION[1]}`;

const output = execSync(
`node createBucketWithDualRegion.js ${bucketNameDualRegion} ${DUAL_REGION[0]} ${DUAL_REGION[1]}`
);

assert.include(output, `${bucketNameDualRegion} created in '${dualRegion}'`);

const [exists] = await dualRegionBucket.exists();
assert.strictEqual(exists, true);

const [metadata] = await dualRegionBucket.getMetadata();
assert.strictEqual(metadata.location, dualRegion);
assert.strictEqual(metadata.locationType, 'dual-region');
});

it('should create a dual-region bucket with turbo replication enabled', async () => {
const output = execSync(
`node createBucketWithTurboReplication.js ${bucketNameDualRegion}`
`node createBucketWithTurboReplication.js ${bucketNameDualRegionTurbo}`
);
assert.match(
output,
new RegExp(
`${bucketNameDualRegion} created with the recovery point objective \\(RPO\\) set to ASYNC_TURBO in NAM4.`
`${bucketNameDualRegionTurbo} created with the recovery point objective \\(RPO\\) set to ASYNC_TURBO in NAM4.`
)
);
const [exists] = await dualRegionBucket.exists();
const [exists] = await dualRegionBucketTurbo.exists();
assert.strictEqual(exists, true);
});

it("should get a bucket's RPO metadata", async () => {
await storage.bucket(bucketNameDualRegion).setMetadata({
await storage.bucket(bucketNameDualRegionTurbo).setMetadata({
rpo: RPO_ASYNC_TURBO,
});

const output = execSync(`node getRPO.js ${bucketNameDualRegion}`);
const output = execSync(`node getRPO.js ${bucketNameDualRegionTurbo}`);
assert.match(
output,
new RegExp(`RPO is ASYNC_TURBO for ${bucketNameDualRegion}.`)
new RegExp(`RPO is ASYNC_TURBO for ${bucketNameDualRegionTurbo}.`)
);

const metadata = await dualRegionBucket.getMetadata();
const metadata = await dualRegionBucketTurbo.getMetadata();
assert.strictEqual(metadata[0].rpo, RPO_ASYNC_TURBO);
});

it("should set a bucket's RPO to ASYNC_TURBO", async () => {
const output = execSync(`node setRPOAsyncTurbo.js ${bucketNameDualRegion}`);
const output = execSync(
`node setRPOAsyncTurbo.js ${bucketNameDualRegionTurbo}`
);
assert.match(
output,
new RegExp(`Turbo replication enabled for ${bucketNameDualRegion}.`)
new RegExp(`Turbo replication enabled for ${bucketNameDualRegionTurbo}.`)
);

const metadata = await dualRegionBucket.getMetadata();
const metadata = await dualRegionBucketTurbo.getMetadata();
assert.strictEqual(metadata[0].rpo, RPO_ASYNC_TURBO);
});

it("should set a bucket's RPO to DEFAULT", async () => {
const output = execSync(`node setRPODefault.js ${bucketNameDualRegion}`);
const output = execSync(`node setRPODefault.js ${bucketNameDualRegionTurbo}`);
assert.match(
output,
new RegExp(`Turbo replication disabled for ${bucketNameDualRegion}.`)
new RegExp(`Turbo replication disabled for ${bucketNameDualRegionTurbo}.`)
);

const metadata = await dualRegionBucket.getMetadata();
const metadata = await dualRegionBucketTurbo.getMetadata();
assert.strictEqual(metadata[0].rpo, RPO_DEFAULT);
});

Expand Down
6 changes: 4 additions & 2 deletions src/storage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -753,15 +753,17 @@ export class Storage extends Service {
* @property {Cors[]} [cors=[]] Specify the CORS configuration to use.
* @property {boolean} [dra=false] Specify the storage class as Durable Reduced
* Availability.
* @property {string} [location] Specify the location / region in which to create the bucket.
* @property {string} [location] Specify the bucket's location(s). If specifying
* a dual-region, can be specified as a string `"US-CENTRAL1+US-WEST1"`.
* For more information, see {@link https://cloud.google.com/storage/docs/locations| Bucket Locations}.
* @property {boolean} [multiRegional=false] Specify the storage class as
* Multi-Regional.
* @property {boolean} [nearline=false] Specify the storage class as Nearline.
* @property {boolean} [regional=false] Specify the storage class as Regional.
* @property {boolean} [requesterPays=false] **Early Access Testers Only**
* Force the use of the User Project metadata field to assign operational
* costs when an operation is made on a Bucket and its objects.
* @property {string} [rpo] For dual region buckets, controls whether turbo
* @property {string} [rpo] For dual-region buckets, controls whether turbo
* replication is enabled (`ASYNC_TURBO`) or disabled (`DEFAULT`).
* @property {boolean} [standard=true] Specify the storage class as Standard.
* @property {string} [storageClass] The new storage class. (`standard`,
Expand Down
23 changes: 23 additions & 0 deletions system-test/storage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -929,6 +929,29 @@ describe('storage', () => {
});
});

describe('dual-region', () => {
let bucket: Bucket;

const REGION1 = 'US-EAST1';
const REGION2 = 'US-WEST1';

beforeEach(() => {
bucket = storage.bucket(generateName());
});

it('creates a dual-region bucket', async () => {
const dualRegion = `${REGION1}+${REGION2}`;
await bucket.create({location: dualRegion});

const [exists] = await bucket.exists();
assert.strictEqual(exists, true);

const [bucketMetadata] = await bucket.getMetadata();
assert.strictEqual(bucketMetadata.location, dualRegion);
assert.strictEqual(bucketMetadata.locationType, 'dual-region');
});
});

describe('uniform bucket-level access', () => {
let bucket: Bucket;

Expand Down