Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix multipart upload - move query parameters to HTTP request query object #47

Merged
merged 4 commits into from
May 9, 2023

Conversation

immavalls
Copy link
Contributor

@immavalls immavalls commented May 8, 2023

Corrects #42, which was not working in AWS s3.

I also corrected some typos in the comments and fixed the protocol to be this.scheme, allowing for either http or https.

@immavalls immavalls requested a review from oleiade May 8, 2023 18:39
Copy link
Member

@oleiade oleiade left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @immavalls 🙇🏻 👍🏻

Could we add the following example in the examples folder please (as s3-multipart.js?)

import crypto from 'k6/crypto'
import exec from 'k6/execution'

import { AWSConfig, S3Client } from '../build/s3.js'

const awsConfig = new AWSConfig({
    region: __ENV.AWS_REGION,
    accessKeyId: __ENV.AWS_ACCESS_KEY_ID,
    secretAccessKey: __ENV.AWS_SECRET_ACCESS_KEY,
    sessionToken: __ENV.AWS_SESSION_TOKEN,
})

const s3 = new S3Client(awsConfig)

const testBucketName = 'test-jslib-aws'
const testFileKey = 'multipart.txt'

export default function () {
    // List the buckets the AWS authentication configuration
    // gives us access to.
    const buckets = s3.listBuckets()

    // If our test bucket does not exist, abort the execution.
    if (buckets.filter((b) => b.name === testBucketName).length == 0) {
        exec.test.abort()
    }

    // Produce random bytes to upload of size ~12MB, that
    // we will upload in two 6MB parts. This is done as the
    // minimum part size supported by S3 is 5MB.
    const bigFile = crypto.randomBytes(12 * 1024 * 1024)

    // Initialize a multipart upload
    const multipartUpload = s3.createMultipartUpload(testBucketName, testFileKey)

    // Upload the first part
    const firstPartData = bigFile.slice(0, 6 * 1024 * 1024)
    const firstPart = s3.uploadPart(
        testBucketName,
        testFileKey,
        multipartUpload.uploadId,
        1,
        firstPartData
    )

    // Upload the second part
    const secondPartData = bigFile.slice(6 * 1024 * 1024, 12 * 1024 * 1024)
    const secondPart = s3.uploadPart(
        testBucketName,
        testFileKey,
        multipartUpload.uploadId,
        2,
        secondPartData
    )

    // Complete the multipart upload
    s3.completeMultipartUpload(testBucketName, testFileKey, multipartUpload.uploadId, [
        firstPart,
        secondPart,
    ])

    // Let's redownload it verify it's correct, and delete it
    const obj = s3.getObject(testBucketName, testFileKey)
    s3.deleteObject(testBucketName, testFileKey)
}

@immavalls
Copy link
Contributor Author

Thanks for the review and the example @oleiade, I just added it to the examples folder, so we might be ready to merge? I will work on finishing the docs part now with the example.

@immavalls immavalls requested a review from oleiade May 9, 2023 15:56
Copy link
Member

@oleiade oleiade left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do it 👏🏻

@immavalls immavalls merged commit adab2d4 into main May 9, 2023
@immavalls immavalls deleted the fix/multipart-upload branch May 9, 2023 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants