"failed to load region from IMDS" back in 0.19 despite AWS_EC2_METADATA_DISABLED=true

[astrojuanlu]

Environment

Delta-rs version:0.19.1

Binding: Python

Environment:

• Cloud provider:
• OS: macOS Sonoma 14.6.1
• Other:

---

Bug

What happened:

minio_storage_options = {
    "AWS_ENDPOINT_URL": "http://127.0.0.1:9010",
    "AWS_ACCESS_KEY_ID": "minioadmin",
    "AWS_SECRET_ACCESS_KEY": "minioadmin",
    "AWS_ALLOW_HTTP": "true",
    "AWS_EC2_METADATA_DISABLED": "true",
    "AWS_S3_ALLOW_UNSAFE_RENAME": "true",
}

df.write_delta("s3://data/bronze_eps", storage_options=minio_storage_options)

yields

[2024-08-23T08:03:23Z WARN  aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
[2024-08-23T08:03:24Z WARN  aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))

What you expected to happen:

As #2377 described, AWS_EC2_METADATA_DISABLED=true used to address this (although I read it became the default at some point?) but it's not the case anymore.

I tested with deltalake==0.18.2 and the same storage_options and the warnings don't appear.

How to reproduce it: (See above)

More details:

[ion-elgreco]

@astrojuanlu fix is already up ;) #2817