Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend Expression.url.upload() to support row-specific URLs using an input column instead of a single prefix #3320

Closed
NellyWhads opened this issue Nov 18, 2024 · 3 comments
Assignees
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed p2 Nice to have features

Comments

@NellyWhads
Copy link

Is your feature request related to a problem?

Currently, users can only upload to a single prefix (local directory, or S3).

The feature request is to extent the upload functionality to work on a column expression.

Describe the solution you'd like

I would like to be able to run this example

df = df.with_column("uploaded_url", df["foo"].url.upload(df["target_urls"]))

This is expected to produce a column named "uploaded_url" which contains paths to successfully uploaded data or null to indicate failure. This would allow API consistency with the download() method, which allows users to optionally raise errors in an eager manner, or ignore them and report a null value from the expression.

Describe alternatives you've considered

I currently maintain a custom StatefulUDF which does the same thing, however, the small API change would remove the need for this solution.

Additional Context

Slack Thread: https://dist-data.slack.com/archives/C041NA2RBFD/p1731900336643709

Would you like to implement a fix?

No

@NellyWhads NellyWhads added enhancement New feature or request needs triage labels Nov 18, 2024
@desmondcheongzx desmondcheongzx self-assigned this Nov 18, 2024
@desmondcheongzx desmondcheongzx added good first issue Good for newcomers p2 Nice to have features and removed needs triage labels Nov 18, 2024
@desmondcheongzx
Copy link
Contributor

FYI aiming to find some time to add this extension towards the end of the week

@ccmao1130 ccmao1130 added the help wanted Extra attention is needed label Nov 26, 2024
@NellyWhads
Copy link
Author

@desmondcheongzx Hello again - any chance there's been progress on this?

desmondcheongzx added a commit that referenced this issue Dec 21, 2024
…ific URLs (#3518)

Addresses #3320

`Expression.url.upload()` can now take in a column of urls to upload
each row to a specific url.
@desmondcheongzx
Copy link
Contributor

Merged in the PR for this, it'll be in the first release of the new year. In the meantime, happy holidays!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed p2 Nice to have features
Projects
None yet
Development

No branches or pull requests

3 participants