Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support Bucket/Object lock operations #374

Merged
merged 1 commit into from
Oct 9, 2018

Conversation

stephenplusplus
Copy link
Contributor

@stephenplusplus stephenplusplus commented Sep 10, 2018

This is a recreation of the original PR: #320. Head there for discussions held during development of this feature.

To Dos

  • Tests
    • System
    • Unit

References

This introduces new behavior:

Buckets

  • bucket#lock()
    Lock a previously-defined retention policy. This will prevent changes to the policy.

  • bucket#removeRetentionPeriod()
    Remove an already-existing retention policy from this bucket.

  • bucket#setRetentionPeriod(durationInSeconds)
    Lock all objects contained in the bucket, based on their creation time.

Files

  • file#getExpirationDate()
    Get a Date object representing the earliest time this file will expire.

Additionally, bucket#getMetadata(), bucket#setMetadata(), file#getMetadata(), and file#setMetadata() are available if users wish to interact with the raw resource schemas as defined by the API.

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Sep 10, 2018
@ghost ghost assigned stephenplusplus Sep 10, 2018
@stephenplusplus stephenplusplus added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Sep 12, 2018
@stephenplusplus
Copy link
Contributor Author

Just a ping @frankyn that this needs an official approval again (nothing has changed since the last PR I accidentally merged). That way, we're good to go when release time comes.

@ghost ghost assigned JustinBeckwith Sep 17, 2018
Copy link
Member

@frankyn frankyn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @stephenplusplus, I think one additional convenience method similar to getExpirationDate() would be helpful for RetentionPolicy.effectiveTime.

@frankyn
Copy link
Member

frankyn commented Sep 19, 2018

Sorry @stephenplusplus, it'd be better to just return the entire retentionPolicy object instead of just the getEffectiveTime() as it also contains isLocked another helpful boolean and the retentionPeriod.

@frankyn frankyn dismissed their stale review September 20, 2018 23:44

Ignore my comments this is fine.

@googlebot
Copy link

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

@googlebot googlebot added cla: no This human has *not* signed the Contributor License Agreement. and removed cla: yes This human has signed the Contributor License Agreement. labels Sep 30, 2018
@JustinBeckwith
Copy link
Contributor

@frankyn @stephenplusplus is this something we can update and merge?

@ghost ghost assigned frankyn Oct 3, 2018
@frankyn frankyn added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Oct 3, 2018
@googlebot
Copy link

A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

@googlebot
Copy link

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

@googlebot googlebot added cla: no This human has *not* signed the Contributor License Agreement. and removed cla: yes This human has signed the Contributor License Agreement. labels Oct 3, 2018
@frankyn frankyn force-pushed the revert-373-revert-320-spp--bucket-lock branch from 1fc4b79 to 6f85575 Compare October 3, 2018 21:53
@stephenplusplus stephenplusplus force-pushed the revert-373-revert-320-spp--bucket-lock branch from 6f85575 to 161dd95 Compare October 3, 2018 22:23
@stephenplusplus
Copy link
Contributor Author

stephenplusplus commented Oct 3, 2018

@JustinBeckwith ! We're trying to fit this change in with the TypeScript changes that have come recently. I'm getting lots of errors trying to run npm test that I thought you might be able to help with:

$ npm t

> @google-cloud/[email protected] test /Users/stephen/dev/nodejs-storage
> npm run test-only


> @google-cloud/[email protected] pretest-only /Users/stephen/dev/nodejs-storage
> npm run compile


> @google-cloud/[email protected] compile /Users/stephen/dev/nodejs-storage
> tsc -p .

system-test/storage.ts:905:14 - error TS2339: Property 'then' does not exist on type 'void'.

905             .then(() => bucket.getMetadata())
                 ~~~~

system-test/storage.ts:919:14 - error TS2339: Property 'then' does not exist on type 'void'.

919             .then(() => bucket.setRetentionPeriod(RETENTION_DURATION_SECONDS))
                 ~~~~

system-test/storage.ts:934:14 - error TS2339: Property 'then' does not exist on type 'void'.

934             .then(() => bucket.setRetentionPeriod(RETENTION_DURATION_SECONDS))
                 ~~~~

system-test/storage.ts:949:14 - error TS2339: Property 'then' does not exist on type 'void'.

949             .then(() => bucket.setRetentionPeriod(RETENTION_DURATION_SECONDS))
                 ~~~~

system-test/storage.ts:971:14 - error TS2339: Property 'then' does not exist on type 'void'.

971             .then(() => FILE.save('data'));
                 ~~~~

system-test/storage.ts:984:14 - error TS2339: Property 'then' does not exist on type 'void'.

984             .then(response => {
                 ~~~~

system-test/storage.ts:1000:14 - error TS2339: Property 'then' does not exist on type 'void'.

1000             .then(response => {
                  ~~~~

system-test/storage.ts:1015:41 - error TS2339: Property 'then' does not exist on type 'void'.

1015         return FILE.getExpirationDate().then(response => {
                                             ~~~~

system-test/storage.ts:1030:20 - error TS2345: Argument of type 'File' is not assignable to parameter of type 'never'.

1030         FILES.push(file);
                        ~~~~

system-test/storage.ts:1044:16 - error TS2339: Property 'setMetadata' does not exist on type 'never'.

1044           file.setMetadata({temporaryHold: null}, err => {
                    ~~~~~~~~~~~

system-test/storage.ts:1050:18 - error TS2339: Property 'delete' does not exist on type 'never'.

1050             file.delete(next);

Any words of wisdom as I stare down these angry lines?

@frankyn frankyn added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Oct 4, 2018
@googlebot
Copy link

A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

@googlebot
Copy link

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

@googlebot googlebot removed the cla: yes This human has signed the Contributor License Agreement. label Oct 4, 2018
@googlebot googlebot added the cla: no This human has *not* signed the Contributor License Agreement. label Oct 4, 2018
@stephenplusplus
Copy link
Contributor Author

I went with just using the callback style for the methods that had issues, and it seems to work now.

System tests work locally for me.

@frankyn
Copy link
Member

frankyn commented Oct 4, 2018

From System tests

 1) storage
       without authentication
         public data
           should list and download a file:
     Uncaught DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
      at emitWarning (internal/process/promises.js:92:15)
      at emitPendingUnhandledRejections (internal/process/promises.js:109:11)
      at process._tickCallback (internal/process/next_tick.js:189:7)
  2) storage
       without authentication
         private data
           should not download a file:
      Uncaught AssertionError [ERR_ASSERTION]: false == true
      + expected - actual
      -false
      +true
      at DestroyableTransform.file.download.err (system-test/storage.ts:159:11)
      at DestroyableTransform.f (node_modules/once/once.js:25:25)
      at /tmpfs/src/github/nodejs-storage/node_modules/through2/through2.js:19:12
      at _combinedTickCallback (internal/process/next_tick.js:131:7)
      at process._tickCallback (internal/process/next_tick.js:180:9)
  3) storage
       without authentication
         private data
           should not upload a file:
     Uncaught TypeError: Cannot read property 'indexOf' of undefined
      at Pumpify.file.save.err (system-test/storage.ts:167:27)
      at Pumpify.Duplexify._destroy (node_modules/duplexify/index.js:191:15)
      at /tmpfs/src/github/nodejs-storage/node_modules/duplexify/index.js:182:10
      at _combinedTickCallback (internal/process/next_tick.js:131:7)
      at process._tickCallback (internal/process/next_tick.js:180:9)
  4) storage
       write, read, and remove files
         stream write
           should resume an upload after an interruption:
     Uncaught Error: write after end
      at writeAfterEnd (node_modules/readable-stream/lib/_stream_writable.js:288:12)
      at Upload.Writable.write (node_modules/readable-stream/lib/_stream_writable.js:332:20)
      at Duplexify._write (node_modules/duplexify/index.js:208:22)
      at doWrite (node_modules/readable-stream/lib/_stream_writable.js:428:64)
      at writeOrBuffer (node_modules/readable-stream/lib/_stream_writable.js:417:5)
      at Duplexify.Writable.write (node_modules/readable-stream/lib/_stream_writable.js:334:11)
      at DestroyableTransform.ondata (node_modules/readable-stream/lib/_stream_readable.js:619:20)
      at addChunk (node_modules/readable-stream/lib/_stream_readable.js:291:12)
      at readableAddChunk (node_modules/readable-stream/lib/_stream_readable.js:278:11)
      at DestroyableTransform.Readable.push (node_modules/readable-stream/lib/_stream_readable.js:245:10)
      at DestroyableTransform.Transform.push (node_modules/readable-stream/lib/_stream_transform.js:148:32)
      at DestroyableTransform.afterTransform (node_modules/readable-stream/lib/_stream_transform.js:91:10)
      at DestroyableTransform.onData [as _transform] (node_modules/hash-stream-validation/index.js:26:5)
      at DestroyableTransform.Transform._read (node_modules/readable-stream/lib/_stream_transform.js:184:10)
      at DestroyableTransform.Transform._write (node_modules/readable-stream/lib/_stream_transform.js:172:83)
      at doWrite (node_modules/readable-stream/lib/_stream_writable.js:428:64)
      at writeOrBuffer (node_modules/readable-stream/lib/_stream_writable.js:417:5)
      at DestroyableTransform.Writable.write (node_modules/readable-stream/lib/_stream_writable.js:334:11)
      at DestroyableTransform.ondata (node_modules/readable-stream/lib/_stream_readable.js:619:20)
      at addChunk (node_modules/readable-stream/lib/_stream_readable.js:291:12)
      at readableAddChunk (node_modules/readable-stream/lib/_stream_readable.js:278:11)
      at DestroyableTransform.Readable.push (node_modules/readable-stream/lib/_stream_readable.js:245:10)
      at DestroyableTransform.Transform.push (node_modules/readable-stream/lib/_stream_transform.js:148:32)
      at DestroyableTransform.afterTransform (node_modules/readable-stream/lib/_stream_transform.js:91:10)
      at DestroyableTransform.noop [as _transform] (node_modules/through2/through2.js:26:3)
      at DestroyableTransform.Transform._read (node_modules/readable-stream/lib/_stream_transform.js:184:10)
      at DestroyableTransform.Transform._write (node_modules/readable-stream/lib/_stream_transform.js:172:83)
      at doWrite (node_modules/readable-stream/lib/_stream_writable.js:428:64)
      at writeOrBuffer (node_modules/readable-stream/lib/_stream_writable.js:417:5)
      at DestroyableTransform.Writable.write (node_modules/readable-stream/lib/_stream_writable.js:334:11)
      at Pumpify.Duplexify._write (node_modules/duplexify/index.js:208:22)
      at doWrite (node_modules/readable-stream/lib/_stream_writable.js:428:64)
      at writeOrBuffer (node_modules/readable-stream/lib/_stream_writable.js:417:5)
      at Pumpify.Writable.write (node_modules/readable-stream/lib/_stream_writable.js:334:11)
      at DestroyableTransform.ondata (node_modules/readable-stream/lib/_stream_readable.js:619:20)
      at addChunk (node_modules/readable-stream/lib/_stream_readable.js:291:12)
      at readableAddChunk (node_modules/readable-stream/lib/_stream_readable.js:278:11)
      at DestroyableTransform.Readable.push (node_modules/readable-stream/lib/_stream_readable.js:245:10)
      at DestroyableTransform.Transform.push (node_modules/readable-stream/lib/_stream_transform.js:148:32)
      at DestroyableTransform._transform (system-test/storage.ts:1766:26)
      at DestroyableTransform.Transform._read (node_modules/readable-stream/lib/_stream_transform.js:184:10)
      at DestroyableTransform.Transform._write (node_modules/readable-stream/lib/_stream_transform.js:172:83)
      at doWrite (node_modules/readable-stream/lib/_stream_writable.js:428:64)
      at writeOrBuffer (node_modules/readable-stream/lib/_stream_writable.js:417:5)
      at DestroyableTransform.Writable.write (node_modules/readable-stream/lib/_stream_writable.js:334:11)
      at ReadStream.ondata (_stream_readable.js:639:20)
      at addChunk (_stream_readable.js:263:12)
      at readableAddChunk (_stream_readable.js:250:11)
      at ReadStream.Readable.push (_stream_readable.js:208:10)
      at fs.read (fs.js:2058:12)
      at FSReqWrap.wrapper [as oncomplete] (fs.js:658:17)

@stephenplusplus
Copy link
Contributor Author

@JustinBeckwith could the system test errors above just be a Kokoro thing? Can anyone verify they're failing locally for them as well?

@frankyn
Copy link
Member

frankyn commented Oct 4, 2018

@stephenplusplus when I run this locally I see the following:

npm install 

system-test/storage.ts:1564:47 - error TS2322: Type 'string | Buffer' is not assignable to type 'Uint8Array'.
  Type 'string' is not assignable to type 'Uint8Array'.

1564                   data = Buffer.concat([data, chunk]);

@frankyn frankyn added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 5, 2018
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 5, 2018
@frankyn frankyn force-pushed the revert-373-revert-320-spp--bucket-lock branch from 995864a to b3e501a Compare October 5, 2018 00:29
@JustinBeckwith JustinBeckwith added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Oct 8, 2018
@googlebot
Copy link

A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

@JustinBeckwith JustinBeckwith changed the title Support Bucket/Object lock operations feat: Support Bucket/Object lock operations Oct 8, 2018
@JustinBeckwith
Copy link
Contributor

@kinwa91 are we close on the system tests for kokoro passing here on master? If not, can you manually verify this PR is gtg?

@JustinBeckwith JustinBeckwith force-pushed the revert-373-revert-320-spp--bucket-lock branch from ae64919 to c39d2e8 Compare October 9, 2018 16:09
@googlebot
Copy link

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

@googlebot googlebot added cla: no This human has *not* signed the Contributor License Agreement. and removed cla: yes This human has signed the Contributor License Agreement. labels Oct 9, 2018
@JustinBeckwith JustinBeckwith force-pushed the revert-373-revert-320-spp--bucket-lock branch from c39d2e8 to 117553f Compare October 9, 2018 16:13
@googlebot
Copy link

CLAs look good, thanks!

@googlebot googlebot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Oct 9, 2018
@JustinBeckwith
Copy link
Contributor

Well @frankyn I'm pretty sure we fixed the system tests and exhausted the GCS bucket create quota 🤣 If you can fix that on long-door-651 this should be gtg. Anything I can do to help:

@JustinBeckwith JustinBeckwith added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 9, 2018
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 9, 2018
@JustinBeckwith JustinBeckwith added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 9, 2018
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 9, 2018
@JustinBeckwith JustinBeckwith merged commit 26e8710 into master Oct 9, 2018
@fhinkel fhinkel deleted the revert-373-revert-320-spp--bucket-lock branch October 12, 2018 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants