-
Notifications
You must be signed in to change notification settings - Fork 591
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
S3.GetObject no longer returns the result as a string #1877
Comments
This happens as
For your specific example, you can write a streamToString function to convert ReadableStream to a string. const { S3Client, GetObjectCommand } = require("@aws-sdk/client-s3");
(async () => {
const region = "us-west-2";
const client = new S3Client({ region });
const streamToString = (stream) =>
new Promise((resolve, reject) => {
const chunks = [];
stream.on("data", (chunk) => chunks.push(chunk));
stream.on("error", reject);
stream.on("end", () => resolve(Buffer.concat(chunks).toString("utf8")));
});
const command = new GetObjectCommand({
Bucket: "test-aws-sdk-js-1877",
Key: "readme.txt",
});
const { Body } = await client.send(command);
const bodyContents = await streamToString(Body);
console.log(bodyContents);
})(); @igilham Does this resolve your query? |
Thanks, @trivikr. This works in my application but raises a few concerns about the library that are worth sharing:
For reference, I've rewritten the import { Readable } from 'stream';
// Apparently the stream parameter should be of type Readable|ReadableStream|Blob
// The latter 2 don't seem to exist anywhere.
async function streamToString (stream: Readable): Promise<string> {
return await new Promise((resolve, reject) => {
const chunks: Uint8Array[] = [];
stream.on('data', (chunk) => chunks.push(chunk));
stream.on('error', reject);
stream.on('end', () => resolve(Buffer.concat(chunks).toString('utf-8')));
});
} |
Documentation for getObject operation lists that
I'm using Visual Studio Code, and it shows type of Please create a new issue with details of your IDE and code if problem persists. |
As this code is run on Node.js, you can pass const bodyContents = await streamToString(Body as Readable); |
Thanks for following up. I didn't realise the methods and types were documented. I took the description on the client landing page (go to the README) to mean it was a dead-end. Perhaps improving the wording should be a separate issue. I can't explain the IDE issue. I'm also on VSCode and it says it's an |
I've created documentation update request at #1878 |
Thanks. I think that covers my remaining frustrations. I appreciate that it can take time for documentation elsewhere to catch up when a major version is released. |
The codesnippet works in Node.js environment, in the browser, you would have a ReadableStream instead of Readable. Here is my implementation of handling the ReadableStream: const streamToString = (stream) => {
return new Promise((resolve, reject) => {
if (stream instanceof ReadableStream === false) {
reject(
"Expected stream to be instance of ReadableStream, but got " +
typeof stream
);
}
let text = "";
const decoder = new TextDecoder("utf-8");
const reader = stream.getReader();
const processRead = ({ done, value }) => {
if (done) {
// resolve promise with chunks
console.log("done");
// resolve(Buffer.concat(chunks).toString("utf8"));
resolve(text);
return;
}
text += decoder.decode(value);
// Not done, keep reading
reader.read().then(processRead);
};
// start read
reader.read().then(processRead);
});
}; |
@trivikr Thanks for that link to the docs! I didn't even know they existed till just now. |
I also wasted lots of time on The new Response(response!.body, {}); This will return a Full example: const s3 = new S3({
region: "us-east-1",
credentials: {
accessKeyId: "replace-it",
secretAccessKey: "replace-it",
},
});
const resp = await s3.getObject({
Bucket: "your-bucket",
Key: "your-object-key",
});
console.info(await new Response(resp.Body, {}).text()) It's quite unfortunate that everybody has to go through these hoops to get the content out of the response though. Especially considering that we have to do type checking with things like |
the use of Response looks as the neatest solution right now, for json and text payloads. |
I've been running into these pain points as well, including Lambda invocation. The payload returned is now a Uint8Array, so it takes a few hoops to get it into a usable format: const payload = JSON.parse(Buffer.from(data.Payload).toString()); Whereas in the previous JS SDK, it was simply: const payload = JSON.parse(data.Payload); I don't understand this new direction with the SDK. I can't say I'm a fan. Maybe @trivikr can weigh in. |
Reopening as lot of customers have raised questions. |
I'm also very confused about how to read S3 Body responses with SDK v3. The SDK documentation for GetObjectCommand does not describe how to do it, and the SDK examples are also missing it (awsdocs/aws-doc-sdk-examples#1677). I would ask the AWS SDK team to include in the SDK a simple way to read S3 Body responses. We don't want to re-implement complicated event handlers and helper functions for this simple purpose every time we use GetObject in a project. In v2 we could just say something like For reference, I was able to do this in Node.js by utilizing node-fetch. I would like something like this be included in AWS SDK.
|
A one-line alternative is to use get-stream package, as posted here: #1096 (comment) I understand the reason for returning a ReadableStream, but a built-in helper method would be nice. Reading the whole body into string in memory is probably good for 99% of cases. If some helper method would be a part of the SDK we could just call it as |
This is now documented in the root readme with an example: https://github.com/kuhe/aws-sdk-js-v3/tree/main#streams You do not need to import import { S3 } from "@aws-sdk/client-s3";
const client = new S3({});
const getObjectResult = await client.getObject({
Bucket: "...",
Key: "...",
});
// env-specific stream with added mixin methods.
const bodyStream = getObjectResult.Body;
// one-time transform.
const bodyAsString = await bodyStream.transformToString();
// throws an error on 2nd call, stream cannot be rewound.
const __error__ = await bodyStream.transformToString(); |
Took two solid years, but hey, we have an official solution.. |
Looks like we need to check for Body being undefined though or else we get import { S3 } from "@aws-sdk/client-s3";
const client = new S3({});
const getObjectResult = await client.getObject({
Bucket: "...",
Key: "...",
});
if (!getObjectResult.Body) {
// handle not found
throw new Error("Object not found");
}
// env-specific stream with added mixin methods.
const bodyStream = getObjectResult.Body;
// one-time transform.
const bodyAsString = await bodyStream.transformToString();
// throws an error on 2nd call, stream cannot be rewound.
const __error__ = await bodyStream.transformToString(); |
With Node 18 introducing Web Streams API will this affect s3 download streams in any way? |
Hey @trivikr! Any updates or official word from your side on this? Haven't heard from you in this thread for over a year and a half, and it's especially relevant with Lambda |
@kuhe any idea how do I get this working with sharp ? It's not taking the string or buffer or stream |
this works for me with
|
Can't we just use the stream instead of converting it? |
I was trying to - no success. If you find a way, keep me posted. |
Okay so after spending few hours I got it right. This way we can pipe our s3 response body into sharp and later use the const getObj = new GetObjectCommand({
Bucket,
Key: objectKey,
});
const s3ImgRes = await s3Client.send(getObj);
const sharpImg = sharp().resize({ width: 500 }).toFormat("webp");
// pipe the body to sharp img
s3ImgRes.Body.pipe(sharpImg);
const putObj = new PutObjectCommand({
Bucket,
Key: `converted/${objectKey.replace(/[a-zA-Z]+$/, "webp")}`,
Body: await sharpImg.toBuffer(),
});
await s3Client.send(putObj); But AWS team please please you need update your docs, I know there is a lot to update but as developer its just so much struggle to use AWS services because of insufficient docs. |
Here is an example of how to download an object from S3 and write that as a file to disk, while keeping it as a stream. This example is typescript targeting node. It seems silly to me if we're going to all this trouble of having a stream coming from AWS that we then convert that to a buffer or string to write to disk. I also agree with the sentiments expressed by others in this thread. It is crazy that getObject has become such a complicated operation in the V3 SDK compared with the V2 SDK and is going to trip many people up for years to come. import type { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import fs from 'node:fs'
import { GetObjectCommand, S3 } from '@aws-sdk/client-s3';
async function downloadFile() {
const s3 = new S3({});
const s3Result = await s3.send(new GetObjectCommand({ Bucket: sourceBucket, Key: sourceKey }));
if (!s3Result.Body) {
throw new Error('received empty body from S3');
}
await pipeline(s3Result.Body as Readable, fs.createWriteStream('/tmp/filedownload.zip'));
} |
Found an easy solution using
|
When I use this, I get error during parsing. Error is Call to transformToString returns the json file contents |
@kesavab This means your // ...
if (dataStr) {
try {
data = JSON.parse(dataStr);
} catch (err) {
console.log(err, dataStr);
}
} |
What's the next step after getting the object string if I want to download the file? Should I convert it into a blob file and return it to the frontend? |
Typescript errors if you try to pipe Body even though it is a stream:
The only way around this is type casting - unless anyone has any other ideas. Will there be a fix for this in a future release version? |
Hi all, It is extremely hard for us to keep track of issues that lay at the end of the issue queue on Github. Since the OP's concern was addressed , and additional concerns were also addressed I feel like we can close this. If you still need help whether it be to report a bug, or to ask a question, please re-engage with us on a new thread / discussion. Sorry for the inconvenience, |
can't thank you enough for this workaround :) |
I found an other solution depending on how you want to get your content from s3 when using Node.js >= v16.7.0 using import consumers from 'node:stream/consumers'
import { S3 } from '@aws-sdk/client-s3'
const s3 = new S3()
const { Body } = await s3.getObject({ Bucket: "your-bucket", Key: "your-object-key" })
// as a buffer
const buffer = await consumers.buffer(Body)
// as a text
const text = await consumers.text(Body)
// as a json
const json = await consumers.json(Body) |
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread. |
Hi people, I just want to let you all know that we have introduced a new feature that allows us to convert the body of a bucket's object to a string by doing the following: import {S3Client, GetObjectCommand} from "@aws-sdk/client-s3";
const client = new S3Client({
region: 'us-east-2'
});
const response = await client.send(new GetObjectCommand({
Bucket: process.env.TEST_BUCKET,
Key: process.env.TEST_KEY
}));
console.log(await response.Body.transformToString('utf-8')); This feature is available after version 3.357.0 Thanks! |
Describe the bug
I'm using the
GetObjectCommand
with anS3Client
to pull a file down from S3. In v2 of the SDK I can writeresponse.Body.toString('utf-8')
to turn the response into a string. In v3 of the SDKresponse.Body
is a complex object that does not seem to expose the result of reading from the socket.It's not clear if the SDK's current behaviour is intentional, but the change in behaviour since v2 is significant and undocumented.
SDK version number
3.1.0
Is the issue in the browser/Node.js/ReactNative?
Node.js
Details of the browser/Node.js/ReactNative version
v12.18.0
To Reproduce (observed behavior)
Expected behavior
It should print the text of the file.
Additional context
data.Body
is a complex object with circular references.Object.keys(data.Body)
returns the following:The text was updated successfully, but these errors were encountered: