-
-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vsis3: incomplete support for container credentials #8858
Comments
@moradology Thanks for the detailed analysis. Could you try the patch in #8859 ? I've only unit tested it as I don't have easy access to a EMR-Serverless environment |
And here I was getting excited about lodging my first gdal PR 🤣 Yeah, I'll test this out! |
OK, so the good news in terms of me not writing it is that I apparently have no taste at all when it comes to C++ Here's the diff I had:
|
oops, I thought that you filed an issue because you didn't feel like doing a PR... I should have asked.
well, your proposal looks equally good. The only difference in mine is that I don't fetch AWS_CONTAINER_CREDENTIALS_RELATIVE_URI if we already get a non empty AWS_CONTAINER_CREDENTIALS_FULL_URI, but that's a micro optimization. |
You're too kind. Sorry this is taking a bit - gotta build your branch on a specific base image, put it on ecr, write a little program for the jvm, etc Let me also say this: libs on amazon linux available via yum are ancient! |
/vsis3/: takes into account AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable (fixes #8858)
[Backport release/3.8] /vsis3/: takes into account AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable (fixes #8858)
…nment variable (fixes OSGeo#8858)
Expected behavior and actual behavior.
I expected
vsis3
to transparently acquire credentials from the environment as it does on EC2 and as generally works on container services provided by AWS (e.g. ECS). On EMR-Serverless, however, this fails to happen.Enabling logging and CURL logging, it becomes obvious that GDAL is failing to connect to the expected metadata/credentials endpoint:
Digging in further, it appears to be the case that this section of code is incomplete as AWS_CONTAINER_CREDENTIALS_FULL_URI is not referenced:
https://github.com/OSGeo/gdal/blob/master/port/cpl_aws.cpp#L835C8-L849
Unfortunately, the documentation doesn't really describe when/where FULL vs RELATIVE environment variables are to be expected. Unhelpfully, the upshot of this documentation is "FULL might be used if RELATIVE isn't".
Reviewing the environment in the cluster I deployed, it becomes clear that this is exactly the issue:
Steps to reproduce the problem.
Try to use GDAL's
vsis3
support on EMR-Serverless. It will 403.Operating system
All operating systems.
GDAL version and provenance
Every single version of GDAL up to and including
master
.How to resolve
The resolution here is relatively simple, I think. cpl_aws.cpp simply needs to have a bit more logic that enables it to use
AWS_CONTAINER_CREDENTIALS_FULL_URI
should that be provided in the container environment.The text was updated successfully, but these errors were encountered: