-
Notifications
You must be signed in to change notification settings - Fork 585
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[client-s3] client is not sending chunks read from fs.ReadSteam when ChecksumAlgorithm is set #3368
Comments
This comment was marked as outdated.
This comment was marked as outdated.
Verified that the chunks read in Repro code: import { S3 } from "@aws-sdk/client-s3"; // v3.53.0
import { createReadStream } from "fs";
const region = "us-west-2";
const client = new S3({ region });
const fileName = "hello-world.txt"; // File contains "helloworld"
// Because of highWaterMark=5, data will be read in two chunks: 'hello', 'world'
const readableStream = createReadStream(fileName, { highWaterMark: 5 });
const Bucket = "aws-sdk-js-trivikr-flexible-checksums-testing"; // replace with your bucket
const Key = fileName;
const Body = readableStream;
const ChecksumAlgorithm = "CRC32";
try {
await client.putObject({ Bucket, Key, Body, ChecksumAlgorithm });
} catch (error) {
console.log({ error });
} console.log added in
Observed behavior: chunk #0: """0
"""
chunk #1: """x-amz-checksum-crc32:+esgrQ==
"""
chunk #2: """
""" Expected behavior: chunk #0: """5
hello
"""
chunk #1: """5
world
"""
chunk #2: """0
"""
chunk #3: """x-amz-checksum-crc32:+esgrQ==
"""
chunk #4: """
""" |
Issue probably related to #3338 |
This issue happens because we pass
This was added in support streams created using filehandle.createReadStream in Node.js >=16.11.0 We'll remove it as of now, and think of handling it later. |
For future reference, this happened as readStream.fd is set in fsCreateReadStream even if it's not set while creating the fs.ReadStream using fs.creatReadStream. The fd gets set when somewhere when middleware stack is getting resolved, as it's not set when resolveMiddleware is called and is set when middleware is called. |
While debugging, I noticed that the valud for fd gets set in Body when resolving endpoint is resolved.
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread. |
Describe the bug
client is not sending chunks read from fs.ReadSteam when ChecksumAlgorithm is set
Your environment
SDK version number
@aws-sdk/[email protected]
Is the issue in the browser/Node.js/ReactNative?
Node.js
Details of the browser/Node.js/ReactNative version
Steps to reproduce
Observed behavior
The data from file is not sent over the wire.
Expected behavior
The request should be successful, and the output should print.
Additional context
Tests for reference:
The text was updated successfully, but these errors were encountered: