Upgrading to 0.31.0 break AWS S3 sink uploads to Cloudflare R2

[ahsandar]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

upgrading to 0.31.0 breaks S3 uploads to R2. Are there any changes to the S3 sink ?

Error

ERROR sink{component_kind="sink" component_id=r2_dns_stream component_type=aws_s3 component_name=r2_dns_stream}:request{request_id=3}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(ServiceError(ServiceError { source: PutObjectError { kind: Unhandled(Unhandled { source: Error { code: Some("NoSuchBucket"), message: Some("The specified bucket does not exist."), request_id: None, extras: {} } }), meta: Error { code: Some("NoSuchBucket"), message: Some("The specified bucket does not exist."), request_id: None, extras: {} } }, raw: Response { inner: Response { status: 404, version: HTTP/1.1, headers: {"date": "Mon, 10 Jul 2023 08:08:02 GMT", "content-type": "application/xml", "content-length": "133", "connection": "keep-alive", "server": "cloudflare", "cf-ray": "7e47510988b92ab0-LAX"}, body: SdkBody { inner: Once(Some(b"<?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist.</Message></Error>")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } })) request_id=3 error_type="request_failed" stage="sending" internal_log_rate_limit=true 

Configuration

[sinks.r2_stream]
  type = "aws_s3"
  inputs = [ "condition.stream" ]
  bucket = "kakashi-stream"
  endpoint = "${R2_ENDPOINT}"
  region = "us-east-1"
  acl = "private"
  filename_append_uuid = true
  filename_extension = "log.gz"
  filename_time_format = "%H_%M_%s"
  key_prefix = "date=%Y%m%d/"
  compression = "gzip"
[sinks.r2_stream.auth]
  access_key_id = "${R2_ACCESS_KEY}"
  secret_access_key = "${R2_SECRET_KEY}"
[sinks.r2_stream.buffer]
  type = "disk"
  when_full = "block"
  max_size = 268435488
[sinks.r2_stream.encoding]
  codec = "json"
[sinks.r2_stream.framing]
  method = "newline_delimited"

Version

0.31.0

Debug Output

No response

Example Data

No response

Additional Context

The aws_s3 sink works well with 0.30.0, upgrading to 0.31.0 throws error and fails to upload any data

References

No response

[jszwedko]

Thanks for opening this @ahsandar ! I'll try it reproduce it, but if you want to do some of your own debugging, you could set:

VECTOR_LOG=aws_smithy_http=trace

To get debug level request/response logs.

[ahsandar]

ok seem to have found the change that worked

config that worked with 0.30.0
R2 S3 endpoint ``https://xxxxxxxxxxxxxxxxxxxxxxxxxxx.r2.cloudflarestorage.com/kakashi-stream`
bucket kakashi-stream

structure in R2
kakashi-stream/kakashi-stream/date=20230710

in 0.31.0 the above doesn't work, but below works if you remove the bucket name from s3 endpoint but then there is no subfolder created by bucket name

R2 S3 endpoint ``https://xxxxxxxxxxxxxxxxxxxxxxxxxxx.r2.cloudflarestorage.com`
bucket kakashi-stream

structure in R2
kakashi-stream/date=20230710

[jszwedko]

Aha, gotcha. This sounds related to some AWS SDK changes for virtual bucket addresses mentioned in https://github.com/awslabs/aws-sdk-rust/blob/main/CHANGELOG.md . Does this resolve the issue well enough for you? We can add a note to the release guide.

[ahsandar]

For me it solves the issue for now, I suppose in case I need a folder in the bucket can use the key prefix to create the path. Something i. the doc or release notes for this would be good