[C++] how to set use_virtual_addressing for pyarrow S3FileSystem

[windpiger]

Currently pyarrow.fs.S3FileSystem init S3Client by using this logic to set use_virtual_addressing :

arrow/cpp/src/arrow/filesystem/s3fs.cc

Line 922 in a2a3b95

const bool use_virtual_addressing = options_.endpoint_override.empty();

arrow/cpp/src/arrow/filesystem/s3fs.cc

Line 956 in a2a3b95

use_virtual_addressing);

But If I want to set endpointOverride to "https://mytest.endpoint" which is not empty, use_virtual_addressing will be set to false because use_virtual_addressing = options_.endpoint_override.empty();

So How can I set use_virtual_addressing to true if I also want to set a non-empty value for endpointOverride?

Maybe we should de-couple these two configs and set them independently

Component(s)

C++

[windpiger]

@pitrou could you pls help me to review this, thank you!

[pitrou]

@windpiger When writing this code, I thought that virtual adressing would be incompatible with a custom endpoint, but it turns out the two should work together (the bucket name would be prepended to the custom hostname: http(s)://<bucket>.endpoint).

[pitrou]

Would you like to submit a PR for this?

[pitrou]

Also cc @felipecrv if you want an entrypoint into the filesystem subsystem.

[windpiger]

Would you like to submit a PR for this?

ok, it's my pleasure~
I have another question that how can I use docker-compose to build a distribution wheel on my MacOS?
docker-compose python-wheel-manylinux-2014 ? @pitrou

[pitrou]

I have another question that how can I use docker-compose to build a distribution wheel on my MacOS?
docker-compose python-wheel-manylinux-2014 ? @pitrou

I have no idea. @raulcd Do you know?

[raulcd]

I have usually reproduced the python wheel generation on our CI with for example:

$ PYTHON=3.11 docker-compose build --no-cache --progress plain python-wheel-manylinux-2014
$ PYTHON=3.11 docker-compose run --rm python-wheel-manylinux-2014

You can also run our wheel tests (once generated) with:

$ docker-compose build --no-cache --progress plain python-wheel-manylinux-test-unittests
$ docker-compose run --rm python-wheel-manylinux-test-unittests

I haven't tested those right now but I have used them in the past.

[windpiger]

thank you, guys, I have done with docker-compose python-wheel-manylinux-2014 @pitrou @raulcd

[mapleFU]

thank you, guys, I have done with docker-compose python-wheel-manylinux-2014

@windpiger would you mind push to github and create a patch?

[pitrou]

This was done in #38858.

[MissiontoMars]

This PR only modifies the C++ layer, so when using the Python API, the "force_virtual_addressing" parameter cannot be passed. such as:

fs = pyarrow.fs.S3FileSystem(access_key='aaa', secret_key='bbb', endpoint_override='https://xxx', force_virtual_addressing=True)
@pitrou

[pitrou]

Yes, sorry. I should have added a separate issue to expose it in Python.