Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(athena): add amazon athena backend #10631

Merged
merged 1 commit into from
Dec 31, 2024
Merged

feat(athena): add amazon athena backend #10631

merged 1 commit into from
Dec 31, 2024

Conversation

cpcloud
Copy link
Member

@cpcloud cpcloud commented Dec 30, 2024

Draft PR for the Amazon Athena backend. Waiting on Voltron Data internal help with CI setup.

@cpcloud cpcloud added this to the 10.0 milestone Dec 30, 2024
@cpcloud cpcloud added the feature Features or general enhancements label Dec 30, 2024
@github-actions github-actions bot added tests Issues or PRs related to tests ci Continuous Integration issues or PRs dependencies Issues or PRs related to dependencies sql Backends that generate SQL nix Issues or PRs related to nix labels Dec 30, 2024
@cpcloud cpcloud force-pushed the athena branch 3 times, most recently from 9d0bf2e to f48b149 Compare December 30, 2024 14:10
@cpcloud
Copy link
Member Author

cpcloud commented Dec 30, 2024

All passing locally:

cloud in 🌐 falcon in …/ibis on  athena is 📦 v9.5.0 via 🐍 v3.12.8 via ❄️  impure (ibis-3.12-env) took 32s
❯ pytest -m athena -n 20 --snapshot-update -q
bringing up nodes...
.ssssssssssssssssssssssssx......ssssssssssssssssssssssss......ssssxxxxx.x.xx.........x....x.........x......... [  5%]
.........xx.........xssssssssssssssssssssssss..........x.....xx..............x...x.x.......xssssssssssssssssss [ 11%]
sssss....xxxx......x.x..........................x..x.x.s.x.x.x.........x...................................... [ 17%]
..xx..x...x.x......xx......x...............x................xxx...x...x.....x....x..x.......xxxxxxxxxx.......x [ 22%]
.......x.......x.........x...........x.....xx.xx.....x...x..............................x..........x.x....xx.. [ 28%]
.x.......x...x....................x...................x..............x...xxx............x............x........ [ 34%]
.....x.......x.x....x........x......x..................x..x.................X.................x...x........x.. [ 39%]
.....................x.....x....................x.............x..........Xx.....................x............. [ 45%]
.......x...x.....x..................xxxx........x....x..xx..............x........x.......x.................... [ 51%]
..x..........x....xxxxxxxxxx...xxxxxxx..x.........sx..................x..x.x.x......x.....x..xxx.............. [ 56%]
.x...x....x....x.......x..x.......x......x...x....s.........x.....x.x..........xsx....xx...x.....x...x...xxx.. [ 62%]
x.x..x.s..ssssssssssssss...........x..x...xx.........xs.........xxx......sssssss....x.x....................... [ 68%]
..x......xx.x.x.s..xx.x..x..x.........x..........xx...x.....x...............x..x......xx....x......xx.xx...x.. [ 73%]
..x.x...x....x..x..xxxxxxxxx.x.....x.x..xx..............x.....xxxxxx...x.x....x.x.x.x.x.....xx....xxx..s.x.x.x [ 79%]
..x...........x..x.xx.x..x.xx.x.x......x..x..x....x.xx...xx.xxxx.x.xx.xx..x.xxxx....x.x.x.x...x........x..xx.. [ 85%]
x.x.........xx.xx...xx.x.x..xxx....xxx.xx..xxx........................................s.s.....x............... [ 91%]
.......................................................xx.................................................x... [ 96%]
....xx.x.x...xx.....x......xxx...x.x.....x..x.....x..x.x.xxx..x.                                               [100%]
1464 passed, 130 skipped, 338 xfailed, 2 xpassed in 546.92s (0:09:06)

@github-actions github-actions bot added the docs Documentation related issues or PRs label Dec 30, 2024
@cpcloud cpcloud added the docs-preview Add this label to trigger a docs preview label Dec 30, 2024
@cpcloud cpcloud requested a review from gforsyth December 30, 2024 14:30
@ibis-docs-bot ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Dec 30, 2024
gforsyth
gforsyth reviewed Dec 30, 2024
View reviewed changes
Copy link
Member

@gforsyth gforsyth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable -- love a cloud-hosted older version of an existing backend with its own quirks.

Only blocker right now is getting the tests running in CI

docs/backends/athena.qmd Show resolved Hide resolved
@cpcloud cpcloud force-pushed the athena branch 2 times, most recently from 7dd1993 to d7ca34e Compare December 31, 2024 12:22
@IndexSeek
Copy link
Member

It's always cool to see a new backend! Is it worth adding a note to the cloud_support_policy?

@cpcloud
Copy link
Member Author

cpcloud commented Dec 31, 2024

It's always cool to see a new backend! Is it worth adding a note to the cloud_support_policy?

Absolutely, good catch!

@cpcloud
Copy link
Member Author

cpcloud commented Dec 31, 2024

Ok, I was able to set up the AWS credentials. I can debug that once it's merged.

@cpcloud cpcloud merged commit 66b3ed2 into ibis-project:main Dec 31, 2024
106 checks passed
@cpcloud cpcloud deleted the athena branch December 31, 2024 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gforsyth gforsyth gforsyth left review comments

Assignees
No one assigned
Labels
ci Continuous Integration issues or PRs dependencies Issues or PRs related to dependencies docs Documentation related issues or PRs feature Features or general enhancements nix Issues or PRs related to nix sql Backends that generate SQL tests Issues or PRs related to tests
Projects
None yet
Milestone
10.0
Development

Successfully merging this pull request may close these issues.
3 participants
@cpcloud @IndexSeek @gforsyth