Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

@uppy/aws-s3: add endpoint option #5173

Merged
merged 23 commits into from
Jun 13, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 19 additions & 12 deletions docs/uploader/aws-s3-multipart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,8 @@ milliseconds on uploading.

**In short**

- We recommend the default value of [`shouldUseMultipart`][], which enable multipart uploads only
for large files.
- We recommend the default value of [`shouldUseMultipart`][], which enable
multipart uploads only for large files.
- If you prefer to have less overhead (+20% upload speed) you can use temporary
S3 credentials with [`getTemporarySecurityCredentials`][]. This means users
get a single token which allows them to do bucket operations for longer,
Expand Down Expand Up @@ -166,7 +166,7 @@ import '@uppy/dashboard/dist/style.min.css';
const uppy = new Uppy()
.use(Dashboard, { inline: true, target: 'body' })
.use(AwsS3, {
companionUrl: 'https://companion.uppy.io',
endpoint: 'https://companion.uppy.io',
});
```

Expand All @@ -179,8 +179,8 @@ const uppy = new Uppy()
A boolean, or a function that returns a boolean which is called for each file
that is uploaded with the corresponding `UppyFile` instance as argument.

By default, all files with a `file.size` ≤ 100 MiB will be uploaded in a single chunk, all files
larger than that as multipart.
By default, all files with a `file.size` ≤ 100 MiB will be uploaded in a
single chunk, all files larger than that as multipart.

Here’s how to use it:

Expand Down Expand Up @@ -215,9 +215,10 @@ uploaded.

:::

#### `companionUrl`
#### `endpoint`

URL to a [Companion](/docs/companion) instance (`string`, default: `null`).
URL to a [Companion](/docs/companion) or Companion-like instance (`string`,
default: `null`).
aduh95 marked this conversation as resolved.
Show resolved Hide resolved

#### `companionHeaders`

Expand Down Expand Up @@ -245,10 +246,10 @@ disable automatic retries, and fail instantly if any chunk fails to upload.
A function that returns the minimum chunk size to use when uploading the given
file as multipart.

For multipart uploads, chunks are sent in batches to
have presigned URLs generated with [`signPart()`](#signpartfile-partdata). To
reduce the amount of requests for large files, you can choose a larger chunk
size, at the cost of having to re-upload more data if one chunk fails to upload.
For multipart uploads, chunks are sent in batches to have presigned URLs
generated with [`signPart()`](#signpartfile-partdata). To reduce the amount of
requests for large files, you can choose a larger chunk size, at the cost of
having to re-upload more data if one chunk fails to upload.

S3 requires a minimum chunk size of 5MiB, and supports at most 10,000 chunks per
multipart upload. If `getChunkSize()` returns a size that’s too small, Uppy will
Expand Down Expand Up @@ -432,7 +433,7 @@ upload sources), you can pass a boolean:
```js
uppy.use(AwsS3, {
// This is an example using Companion:
companionUrl: 'http://companion.uppy.io',
endpoint: 'http://companion.uppy.io',
getTemporarySecurityCredentials: true,
shouldUseMultipart: (file) => file.size > 100 * 2 ** 20,
});
Expand All @@ -454,6 +455,12 @@ uppy.use(AwsS3, {
});
```

#### `companionUrl`

Alias for [`endpoint`](#endpoint).

</details>

[`gettemporarysecuritycredentials`]: #gettemporarysecuritycredentialsoptions
[`shouldusemultipart`]: #shouldusemultipartfile
[companion docs]: /docs/companion
Expand Down
4 changes: 3 additions & 1 deletion packages/@uppy/aws-s3/src/index.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ describe('AwsS3Multipart', () => {
core.use(AwsS3Multipart)
const awsS3Multipart = core.getPlugin('AwsS3Multipart')!

const err = 'Expected a `companionUrl` option'
const err = 'Expected a `endpoint` option'
const file = {}
const opts = {}

Expand Down Expand Up @@ -337,6 +337,7 @@ describe('AwsS3Multipart', () => {

it('companionHeader is updated before uploading file', async () => {
awsS3Multipart.setOptions({
endpoint: 'http://localhost',
companionHeaders: {
authorization: newToken,
},
Expand Down Expand Up @@ -368,6 +369,7 @@ describe('AwsS3Multipart', () => {
Body
>
awsS3Multipart.setOptions({
endpoint: 'http://localhost',
companionHeaders: {
authorization: newToken,
},
Expand Down
62 changes: 47 additions & 15 deletions packages/@uppy/aws-s3/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -142,8 +142,15 @@
ETag?: string
}

type AWSS3WithCompanion = {
companionUrl: string
type AWSS3WithCompanion = (
| {
/** @deprecated use `endpoint` instead */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm okay with doing a breaking change. If not now then when?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the next major

companionUrl: string
}
| {
endpoint: string
}
) & {
companionHeaders?: Record<string, string>
companionCookiesRule?: string
getTemporarySecurityCredentials?: true
Expand Down Expand Up @@ -337,8 +344,7 @@
// We need the `as any` here because of the dynamic default options.
this.type = 'uploader'
this.id = this.opts.id || 'AwsS3Multipart'
// TODO: only initiate `RequestClient` is `companionUrl` is defined.
this.#client = new RequestClient(uppy, opts as any)
this.#setClient(opts)

const dynamicDefaultOptions = {
createMultipartUpload: this.createMultipartUpload,
Expand Down Expand Up @@ -388,9 +394,36 @@
return this.#client
}

#setClient(opts?: Partial<AwsS3MultipartOptions<M, B>>) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think this is a lot of code for backwards compatibility which we're not obliged to do. People can not blindly upgrade Uppy anyway, so you have to already check all things of the migration guide anyway. Why not just errors in the constructor, saying the properties are renamed?

Maybe we should let @mifi settle it, backwards compatible or breaking for 4.x.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are breaking many other apis in 4.x, so why not break this too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not have any backward compatiblity, it's just emitting warnings if folks are using the old option names.

if (
opts == null ||
!(
'endpoint' in opts ||
'companionUrl' in opts ||
'companionHeaders' in opts ||
'companionCookiesRule' in opts ||
'companionKeysParams' in opts
Murderlon marked this conversation as resolved.
Show resolved Hide resolved
)
)
return
if (opts.endpoint) {

Check failure on line 409 in packages/@uppy/aws-s3/src/index.ts

View workflow job for this annotation

GitHub Actions / Type tests

Property 'endpoint' does not exist on type 'Partial<AwsS3MultipartOptions<M, B>>'.

Check failure on line 409 in packages/@uppy/aws-s3/src/index.ts

View workflow job for this annotation

GitHub Actions / Type tests

Property 'endpoint' does not exist on type 'Partial<AwsS3MultipartOptions<M, B>>'.

Check failure on line 409 in packages/@uppy/aws-s3/src/index.ts

View workflow job for this annotation

GitHub Actions / Browser tests

Property 'endpoint' does not exist on type 'Partial<AwsS3MultipartOptions<M, B>>'.

Check failure on line 409 in packages/@uppy/aws-s3/src/index.ts

View workflow job for this annotation

GitHub Actions / Browser tests

Property 'endpoint' does not exist on type 'Partial<AwsS3MultipartOptions<M, B>>'.
// eslint-disable-next-line no-param-reassign
opts = { ...this.opts, companionUrl: opts.endpoint }

Check failure on line 411 in packages/@uppy/aws-s3/src/index.ts

View workflow job for this annotation

GitHub Actions / Type tests

Property 'endpoint' does not exist on type 'Partial<AwsS3MultipartOptions<M, B>>'.

Check failure on line 411 in packages/@uppy/aws-s3/src/index.ts

View workflow job for this annotation

GitHub Actions / Type tests

Property 'endpoint' does not exist on type 'Partial<AwsS3MultipartOptions<M, B>>'.

Check failure on line 411 in packages/@uppy/aws-s3/src/index.ts

View workflow job for this annotation

GitHub Actions / Browser tests

Property 'endpoint' does not exist on type 'Partial<AwsS3MultipartOptions<M, B>>'.

Check failure on line 411 in packages/@uppy/aws-s3/src/index.ts

View workflow job for this annotation

GitHub Actions / Browser tests

Property 'endpoint' does not exist on type 'Partial<AwsS3MultipartOptions<M, B>>'.
} else if (opts.companionUrl) {
this.uppy.log(
'`companionUrl` option is deprecated in @uppy/aws-s3, use `endpoint` instead.',
'warning',
)
// eslint-disable-next-line no-param-reassign
opts = this.opts
}
this.#client = new RequestClient(this.uppy, opts as any)
}

setOptions(newOptions: Partial<AwsS3MultipartOptions<M, B>>): void {
this.#companionCommunicationQueue.setOptions(newOptions)
super.setOptions(newOptions)
this.#setClient(newOptions)
this.#setCompanionHeaders()
}

Expand Down Expand Up @@ -418,11 +451,10 @@
}
}

// TODO: make this a private method in the next major
assertHost(method: string): void {
if (!this.opts.companionUrl) {
#assertHost(method: string): void {
if (!this.#client) {
throw new Error(
`Expected a \`companionUrl\` option containing a Companion address, or if you are not using Companion, a custom \`${method}\` implementation.`,
`Expected a \`endpoint\` option containing a URL, or if you are not using Companion, a custom \`${method}\` implementation.`,
)
}
}
Expand All @@ -431,7 +463,7 @@
file: UppyFile<M, B>,
signal?: AbortSignal,
): Promise<UploadResult> {
this.assertHost('createMultipartUpload')
this.#assertHost('createMultipartUpload')
throwIfAborted(signal)

const allowedMetaFields = getAllowedMetaFields(
Expand Down Expand Up @@ -459,7 +491,7 @@
oldSignal?: AbortSignal,
): Promise<AwsS3Part[]> {
signal ??= oldSignal // eslint-disable-line no-param-reassign
this.assertHost('listParts')
this.#assertHost('listParts')
throwIfAborted(signal)

const filename = encodeURIComponent(key)
Expand All @@ -474,7 +506,7 @@
oldSignal?: AbortSignal,
): Promise<B> {
signal ??= oldSignal // eslint-disable-line no-param-reassign
this.assertHost('completeMultipartUpload')
this.#assertHost('completeMultipartUpload')
throwIfAborted(signal)

const filename = encodeURIComponent(key)
Expand All @@ -496,7 +528,7 @@
if (this.#cachedTemporaryCredentials == null) {
// We do not await it just yet, so concurrent calls do not try to override it:
if (this.opts.getTemporarySecurityCredentials === true) {
this.assertHost('getTemporarySecurityCredentials')
this.#assertHost('getTemporarySecurityCredentials')
this.#cachedTemporaryCredentials = this.#client
.get<AwsS3STSResponse>('s3/sts', options)
.then(assertServerError)
Expand Down Expand Up @@ -559,7 +591,7 @@
file: UppyFile<M, B>,
{ uploadId, key, partNumber, signal }: SignPartOptions,
): Promise<AwsS3UploadParameters> {
this.assertHost('signPart')
this.#assertHost('signPart')
throwIfAborted(signal)

if (uploadId == null || key == null || partNumber == null) {
Expand All @@ -584,7 +616,7 @@
oldSignal?: AbortSignal, // TODO: remove in next major
): Promise<void> {
signal ??= oldSignal // eslint-disable-line no-param-reassign
this.assertHost('abortMultipartUpload')
this.#assertHost('abortMultipartUpload')

const filename = encodeURIComponent(key)
const uploadIdEnc = encodeURIComponent(uploadId)
Expand Down Expand Up @@ -922,7 +954,7 @@
}

#setCompanionHeaders = () => {
this.#client.setCompanionHeaders(this.opts.companionHeaders)
this.#client?.setCompanionHeaders(this.opts.companionHeaders)
}

#setResumableUploadsCapability = (boolean: boolean) => {
Expand Down
Loading