Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deps: update ibis to version 8.0.0 and refactor remote_function to use ibis UDF method #277

Merged
merged 44 commits into from
Feb 28, 2024
Merged

deps: update ibis to version 8.0.0 and refactor remote_function to use ibis UDF method #277

merged 44 commits into from
Feb 28, 2024

Conversation

tswast
Copy link
Collaborator

@tswast tswast commented Dec 15, 2023

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Follow-up to #53 (comment)
🦕

tswast added 25 commits September 22, 2023 16:04
@tswast
deps: migrate to ibis-framework >= "7.0.0"
8929fef 
This should unlock some bug fixes as well as potential `UNNEST` support
in a future change.
@tswast
use dtype instead of output_dtype in custom ops
a4e995d
@tswast
adjust type annotations
8fd8fd4
@tswast
Merge remote-tracking branch 'origin/main' into b301459557-ibis-prere…
ba3c5c4 
…lease
@tswast
Merge branch 'main' into b301459557-ibis-prerelease
f1ce176
@tswast
Update noxfile.py
14fb7c8
@tswast
Merge branch 'main' into b301459557-ibis-prerelease
ea8d3e5
@tswast
Merge remote-tracking branch 'origin/main' into b301459557-ibis-prere…
191df69 
…lease
@tswast
update type annotations
b182453
@tswast
Merge remote-tracking branch 'origin/main' into b301459557-ibis-prere…
174e735 
…lease
@tswast
fix for deferred values
0ab94cc
@tswast
fix prerelease
4931923
@tswast
minimum 7.1.0
9df0816
@tswast
mypy
71f4889
@tswast
revert presubmit changes
6c64ec5
@tswast
fix minimum sqlglot
e37571c
@tswast
fix custom op
fad36a8
@tswast
Merge remote-tracking branch 'origin/main' into b301459557-ibis-prere…
5770502 
…lease
@tswast
hack InMemoryTable formatter back in
c318b18
@tswast
use ops module to avoid breaking changes if ops move around
d3304b2
@tswast
workaround nullscalar issue
33bd2e0
@tswast
update usage of percent_rank to explicitly order by the value
c373dc0
@tswast
disable ibis prerelease tests for now
41364b5
@tswast
fix unit_prerelease
7a8784c
@tswast
refactor: use ibis UDF functionality for remote_function
7b304a9
@product-auto-label product-auto-label bot added size: l Pull request size is large. api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. labels Dec 15, 2023
@tswast tswast mentioned this pull request Dec 15, 2023
Merged
4 tasks
@product-auto-label product-auto-label bot removed the size: l Pull request size is large. label Dec 15, 2023
@tswast tswast changed the title deps: update ibis to version 7.2.0 and refactor remote_function to use ibis UDF method deps: update ibis to version 8.0.0 and refactor remote_function to use ibis UDF method Feb 26, 2024
@tswast
Copy link
Collaborator Author

tswast commented Feb 26, 2024

Looks like there are some changes I need to make to the remote functions type inference to account for the ibis 8.0.0 upgrade.

The nlargest / nsmallest tests are still failing though. I think I need to bump the minimum sqlglot: https://github.com/tobymao/sqlglot/blob/main/CHANGELOG.md#v2080---2024-01-08

@tswast tswast mentioned this pull request Feb 26, 2024
Closed
1 task
@tswast tswast marked this pull request as ready for review February 27, 2024 18:14
@tswast tswast requested review from a team as code owners February 27, 2024 18:14
@tswast tswast requested a review from Genesis929 February 27, 2024 18:14
@tswast tswast added the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 27, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 27, 2024
@tswast tswast added the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 27, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 27, 2024
@tswast tswast added the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 27, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 27, 2024
chelsea-lin
chelsea-lin approved these changes Feb 28, 2024
View reviewed changes
third_party/bigframes_vendored/ibis/backends/bigquery/datatypes.py
@@ -0,0 +1,176 @@
# Contains code from
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious why we need a copy of these codes. Any customization for BigFrames or any known issues in ibis to be worked around?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ibis now maps to the sqlglot type aliases, but we still needed the final BigQuery type for some remote functions code.

@tswast tswast added the automerge Merge the pull request once unit tests and other checks pass. label Feb 28, 2024
@gcf-merge-on-green gcf-merge-on-green bot merged commit 350499b into main Feb 28, 2024
14 of 15 checks passed
@gcf-merge-on-green gcf-merge-on-green bot deleted the tswast-ibis-udf branch February 28, 2024 16:34
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Feb 28, 2024
@release-please release-please bot mentioned this pull request Feb 28, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chelsea-lin chelsea-lin chelsea-lin approved these changes

@Genesis929 Genesis929 Awaiting requested review from Genesis929 Genesis929 is a code owner automatically assigned from googleapis/api-bigquery-dataframe

Assignees
No one assigned
Labels
api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. size: l Pull request size is large.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tswast @chelsea-lin