Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add JSON type compiler support #1130

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

SandeepAkode
Copy link

@SandeepAkode SandeepAkode commented Oct 30, 2024

Fixes #399 🦕

This commit handles JSON as column type. Currently it throws error for that and we can't use str() method on that column like column.type.

@SandeepAkode SandeepAkode requested review from a team as code owners October 30, 2024 09:39
@product-auto-label product-auto-label bot added the size: xs Pull request size is extra small. label Oct 30, 2024
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. label Oct 30, 2024
@chalmerlowe
Copy link
Collaborator

chalmerlowe commented Oct 30, 2024

Would it be possible for you to provide an example output based on having this new code?

i.e. being able to write sqlalchemy-based Python code with this new capability would allow us to generate SQL that looks like what?

A simple example would suffice so that I can quickly picture what we are trying to achieve that isn't currently possible?

I am in the middle of a couple of hot tasks, so I won't be able to come back to this for a day or two, but I am interested in getting this pushed through.

@chalmerlowe chalmerlowe self-assigned this Oct 30, 2024
@chalmerlowe chalmerlowe added kokoro:run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Oct 30, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Oct 30, 2024
@yoshi-kokoro yoshi-kokoro removed the kokoro:run Add this label to force Kokoro to re-run the tests. label Oct 30, 2024
@SandeepAkode
Copy link
Author

SandeepAkode commented Oct 30, 2024

Here's an example. I created a customers_info column

ALTER TABLE
  `test_db.test_table` ADD COLUMN customer_info STRUCT<preferences JSON>; 

I created a simple sqlalchemy engine and inspector. Then I got columns using inspector.get_columns('test_db.test_table')
When we check the type of the column we added

>>> column['type']
STRUCT(preferences=NullType())

Ideally it should be STRUCT(preferences=JSON())

Also, if we try to use str(column['type']) it will throw an error

sqlalchemy.exc.CompileError: Can't generate DDL for NullType(); did you forget to specify a type on this Column?

This is required as we might need to store column type as a string.

Thanks.

@chalmerlowe chalmerlowe added kokoro:run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Nov 13, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 13, 2024
@yoshi-kokoro yoshi-kokoro removed the kokoro:run Add this label to force Kokoro to re-run the tests. label Nov 13, 2024
@chalmerlowe
Copy link
Collaborator

@SandeepAkode

Thank you very much for your interest in this project and for your suggested change. Before we can approve and merge this PR, we will need to add tests. Is that something you are comfortable doing? If not that is fine, we have several options. If you want to add the tests but are not sure what that entails, we can provide guidance along the way. If you really don't feel comfortable, then let us know and we can put this into our backlog of tasks.

@chalmerlowe
Copy link
Collaborator

This is failing this test:

File "/tmpfs/src/github/python-bigquery-sqlalchemy/tests/unit/test_dialect_types.py", line 30, in test_types_import
    assert getattr(dialect_module, type_name) == type_value
AttributeError: module 'sqlalchemy_bigquery' has no attribute 'JSON'|

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. size: xs Pull request size is extra small.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for JSON datatype
4 participants