Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow S3Paths to be stored in Arrow files #184

Closed
haberdashPI opened this issue Aug 11, 2021 · 4 comments · Fixed by #231
Closed

Allow S3Paths to be stored in Arrow files #184

haberdashPI opened this issue Aug 11, 2021 · 4 comments · Fixed by #231

Comments

@haberdashPI
Copy link
Contributor

It would be convenient if S3Paths would seamlessly serialize to Arrow files (via Arrow.jl).

As described by @ericphanson:

We’d want to PR AWSS3 to add a dependency on ArrowTypes.jl (which is very light) and then setup custom serialization (https://arrow.juliadata.org/dev/manual/#Custom-types) to serialize out an S3Path as a string, and to load it back from a string as an S3Path

@haberdashPI
Copy link
Contributor Author

(FYI: happy to try to attempt a PR for this; might be able to get a draft started later today)

@omus
Copy link
Member

omus commented Aug 11, 2021

I think #175 is a prerequisite of this. We definitely don't want to be serializing the config information for Arrow

@ericphanson
Copy link
Member

ericphanson commented Aug 11, 2021

My thought is we can just ignore the config and serialize it out as a string like "s3://bucket/key" (with a type tag calling it an S3Path), and then to deserialize, just parse it back (with the global config at parse time). So it wouldn't need #175

@omus
Copy link
Member

omus commented Aug 11, 2021

The reason I think we should require #175 is that I think we should only perform Arrow serialization on S3Paths that don't include a explicit configuration.

bors bot added a commit that referenced this issue Feb 7, 2022
231: enable S3Paths to be (de)serialized to/from Arrow r=omus a=jrevels

closes #184 

This adds ArrowTypes as a lightweight dependency, which is hopefully okay. 

Co-authored-by: Jarrett Revels <[email protected]>
Co-authored-by: Curtis Vogt <[email protected]>
bors bot added a commit that referenced this issue Feb 7, 2022
231: enable S3Paths to be (de)serialized to/from Arrow r=omus a=jrevels

closes #184 

This adds ArrowTypes as a lightweight dependency, which is hopefully okay. 

Co-authored-by: Jarrett Revels <[email protected]>
Co-authored-by: Curtis Vogt <[email protected]>
bors bot added a commit that referenced this issue Feb 7, 2022
231: enable S3Paths to be (de)serialized to/from Arrow r=omus a=jrevels

closes #184 

This adds ArrowTypes as a lightweight dependency, which is hopefully okay. 

Co-authored-by: Jarrett Revels <[email protected]>
Co-authored-by: Curtis Vogt <[email protected]>
bors bot added a commit that referenced this issue Feb 7, 2022
231: enable S3Paths to be (de)serialized to/from Arrow r=omus a=jrevels

closes #184 

This adds ArrowTypes as a lightweight dependency, which is hopefully okay. 

Co-authored-by: Jarrett Revels <[email protected]>
Co-authored-by: Curtis Vogt <[email protected]>
bors bot added a commit that referenced this issue Feb 7, 2022
231: enable S3Paths to be (de)serialized to/from Arrow r=omus a=jrevels

closes #184 

This adds ArrowTypes as a lightweight dependency, which is hopefully okay. 

Co-authored-by: Jarrett Revels <[email protected]>
Co-authored-by: Curtis Vogt <[email protected]>
bors bot added a commit that referenced this issue Feb 7, 2022
231: enable S3Paths to be (de)serialized to/from Arrow r=omus a=jrevels

closes #184 

This adds ArrowTypes as a lightweight dependency, which is hopefully okay. 

Co-authored-by: Jarrett Revels <[email protected]>
Co-authored-by: Curtis Vogt <[email protected]>
@omus omus closed this as completed in #231 Feb 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants