Assume role support has been broken since 2022 🤣

[oleh-motrunych]

kafka-delta-ingest version 0.2.0 fails to assume the specified IAM role for cross-account writes to S3, resulting in an Access Denied (403) error, the previous version of the application worked correctly under the same configuration.

error log

2024-09-04T18:23:33 [ERROR] - my.app: Ingest service exited with error Writer { source: Storage { source: Generic { store: "S3", source: Client { status: 403, body: Some("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>CAE9GFWZPREMDXXX</RequestId><HostId>fsdfsdfsdfAaXXXXXqUenLk4JxzSkdGI/msdfsfsfdHyg=</HostId></Error>") } } } }

Current setup:
• AWS Account A: kafka-delta-ingest running as an ECS task.
• AWS Account B: S3 bucket for storage and DynamoDB for lock table.

Steps Taken:

1.	With AWS_S3_ASSUME_ROLE_ARN set: The error still occurred with the same 403 Access Denied response.
2.	Without AWS_S3_ASSUME_ROLE_ARN: The error persisted, Same result.
3.	With a dummy IAM role in AWS_S3_ASSUME_ROLE_ARN: Same result.
4.	With S3 in the same account (Account A): Everything worked perfectly, confirming the issue is specific to cross-account role assumption.
5.	Using the old version: Previous versions of kafka-delta-ingest (before v0.2.0) work fine with the exact same IAM role assumption setup.

Expected Behavior:

The application should properly assume the IAM role specified in the AWS_S3_ASSUME_ROLE_ARN environment variable and perform cross-account writes to S3 in Account B.

[rtyler]

Comparing the current main from kafka-delta-ingest to older deployed versions, I have a hunch this is really a delta-io/delta-rs bug and will try to focus there. 🤔

[rtyler]

I believe this happened when we switched from the rusoto crate to the official AWS SDK crate in delta-rs. Those options are still defined in our code, but don't actually get pulled through to configure the underlying AWS client 🤕

[rtyler]

Based on my analysis this change was responsible for removing assume role support. In essence object_store does not have native assume role support and in order to support this functionality we must first assume the role, get the temporary credentials, and then construct the appropriate AmazonS3 object store