[axum 0.7] Streaming from body to S3

[dsferruzza]

Hi!

I have a HTTP endpoint (built using axum) which is supposed (among other things) to take the request's body and store it in S3.
I would like to avoid storing the whole body in memory and instead stream it from the request's body to S3.

With axum 0.6 and aws-sdk-s3 0.34 I managed to make it work using something like this:

// `body` is extracted using `body: axum::extract::BodyStream` in handler's parameters

let body = axum::body::Body::wrap_stream(body);
let byte_stream = aws_sdk_s3::primitives::ByteStream::new(body.into());
let put = my_client
	.s3
	.put_object()
	.bucket(/* ... */)
	.key(/* ... */)
	.content_type(/* ... */)
	.content_length(/* ... */)
	.body(byte_stream)
	.send()
	.await;

I updated both crates to their latest versions, but I could not manage to fix my code.
I don't see any way to create a ByteStream (or a streamed SdkBody) from an axum's Body; am I missing something?

1. I could collect all bytes from the body into a Vec, but I would like to avoid doing that as body can be very large.
2. There may be a solution using ByteStream::from_body_0_4, but the point is to use axum 0.7 which uses http-body 1.0.

[jdisanti]

It sounds like this feature request will solve this: #977

Unfortunately, it doesn't look like there's a workaround until that's implemented. Actually, it looks like you can use the tower-hyper-body-compat crate to convert Axum's http 1.x body into the http 0.4 body that the SDK needs as a workaround.

[rcoh]

I just opened smithy-lang/smithy-rs#3300 — when this is released (sometime next week, most likely), you'll be able to use axum 0.7 again!

[rcoh]

A side note—This may not the best way to stream a request to S3! I think you MAY get better performance by chunking the upload into parts and converting it into the multi-part upload API. That also makes it possible to retry and handle much larger files.