fix: Handle unknown postgres source types gracefully #3634
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
msnidal
changed the title
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: adchia, msnidal The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
msnidal
force-pushed
the
postgres-contrib-ingestion-type-handling
branch
from
48ee3e9
to
3a31655
Compare
msnidal
requested review from
DvirDukhan,
niklasvm and
kevjumba
as code owners
msnidal
force-pushed
the
postgres-contrib-ingestion-type-handling
branch
from
eb30686
to
9d540fd
Compare
Signed-off-by: Mark Snidal <[email protected]>
msnidal
force-pushed
the
postgres-contrib-ingestion-type-handling
branch
from
9d540fd
to
dfb2d6b
Compare
|
Hey, sorry its been ages but I just got back to this, my rebase got a bit messy and had to sign hence the force pushes (no changes to the commit itself) but I managed to get up to date with main, let me know if this looks alright! |
|
/ok-to-test |
adchia
pushed a commit
that referenced
this pull request
Sep 7, 2023
# [0.34.0](v0.33.0...v0.34.0) (2023-09-07) ### Bug Fixes * Add NUMERIC to bq_to_feast type map ([#3719](#3719)) ([6474b4b](6474b4b)) * Fix python unit tests ([#3734](#3734)) ([e81684d](e81684d)) * Handle unknown postgres source types gracefully ([#3634](#3634)) ([d7041f4](d7041f4)) * Pin protobuf version to avoid seg fault on some machines ([028cc20](028cc20)) * Remove unwanted excessive splitting of gcs path, so expected gcs parquet paths are returned from BigQueryRetrievalJob.to_remote_storage() ([#3730](#3730)) ([f2c5988](f2c5988)) * Run store.plan() only when need it. ([#3708](#3708)) ([7bc7c47](7bc7c47)) * Saved datasets no longer break CLI registry-dump command ([#3717](#3717)) ([f28ccc2](f28ccc2)) * Update py3.8 ci requirements for cython 3.0 release ([#3735](#3735)) ([1695c13](1695c13)) ### Features * Enhance customization of Trino connections when using Trino-based Offline Stores ([#3699](#3699)) ([ed7535e](ed7535e)) * Implement gRPC server to ingest streaming features ([#3687](#3687)) ([a3fcd1f](a3fcd1f))
james-crabtree-sp
pushed a commit
to sailpoint/feast
that referenced
this pull request
Sep 14, 2023
Handle more pg types gracefully Signed-off-by: Mark Snidal <[email protected]>
james-crabtree-sp
pushed a commit
to sailpoint/feast
that referenced
this pull request
Sep 14, 2023
# [0.34.0](feast-dev/feast@v0.33.0...v0.34.0) (2023-09-07) ### Bug Fixes * Add NUMERIC to bq_to_feast type map ([feast-dev#3719](feast-dev#3719)) ([6474b4b](feast-dev@6474b4b)) * Fix python unit tests ([feast-dev#3734](feast-dev#3734)) ([e81684d](feast-dev@e81684d)) * Handle unknown postgres source types gracefully ([feast-dev#3634](feast-dev#3634)) ([d7041f4](feast-dev@d7041f4)) * Pin protobuf version to avoid seg fault on some machines ([028cc20](feast-dev@028cc20)) * Remove unwanted excessive splitting of gcs path, so expected gcs parquet paths are returned from BigQueryRetrievalJob.to_remote_storage() ([feast-dev#3730](feast-dev#3730)) ([f2c5988](feast-dev@f2c5988)) * Run store.plan() only when need it. ([feast-dev#3708](feast-dev#3708)) ([7bc7c47](feast-dev@7bc7c47)) * Saved datasets no longer break CLI registry-dump command ([feast-dev#3717](feast-dev#3717)) ([f28ccc2](feast-dev@f28ccc2)) * Update py3.8 ci requirements for cython 3.0 release ([feast-dev#3735](feast-dev#3735)) ([1695c13](feast-dev@1695c13)) ### Features * Enhance customization of Trino connections when using Trino-based Offline Stores ([feast-dev#3699](feast-dev#3699)) ([ed7535e](feast-dev@ed7535e)) * Implement gRPC server to ingest streaming features ([feast-dev#3687](feast-dev#3687)) ([a3fcd1f](feast-dev@a3fcd1f))
james-crabtree-sp
pushed a commit
to sailpoint/feast
that referenced
this pull request
Sep 14, 2023
Handle more pg types gracefully Signed-off-by: Mark Snidal <[email protected]>
james-crabtree-sp
pushed a commit
to sailpoint/feast
that referenced
this pull request
Sep 14, 2023
# [0.34.0](feast-dev/feast@v0.33.0...v0.34.0) (2023-09-07) ### Bug Fixes * Add NUMERIC to bq_to_feast type map ([feast-dev#3719](feast-dev#3719)) ([6474b4b](feast-dev@6474b4b)) * Fix python unit tests ([feast-dev#3734](feast-dev#3734)) ([e81684d](feast-dev@e81684d)) * Handle unknown postgres source types gracefully ([feast-dev#3634](feast-dev#3634)) ([d7041f4](feast-dev@d7041f4)) * Pin protobuf version to avoid seg fault on some machines ([028cc20](feast-dev@028cc20)) * Remove unwanted excessive splitting of gcs path, so expected gcs parquet paths are returned from BigQueryRetrievalJob.to_remote_storage() ([feast-dev#3730](feast-dev#3730)) ([f2c5988](feast-dev@f2c5988)) * Run store.plan() only when need it. ([feast-dev#3708](feast-dev#3708)) ([7bc7c47](feast-dev@7bc7c47)) * Saved datasets no longer break CLI registry-dump command ([feast-dev#3717](feast-dev#3717)) ([f28ccc2](feast-dev@f28ccc2)) * Update py3.8 ci requirements for cython 3.0 release ([feast-dev#3735](feast-dev#3735)) ([1695c13](feast-dev@1695c13)) ### Features * Enhance customization of Trino connections when using Trino-based Offline Stores ([feast-dev#3699](feast-dev#3699)) ([ed7535e](feast-dev@ed7535e)) * Implement gRPC server to ingest streaming features ([feast-dev#3687](feast-dev#3687)) ([a3fcd1f](feast-dev@a3fcd1f))
zseta
pushed a commit
to zseta/feast
that referenced
this pull request
Feb 7, 2024
Handle more pg types gracefully Signed-off-by: Mark Snidal <[email protected]> Signed-off-by: Attila Toth <[email protected]>
zseta
pushed a commit
to zseta/feast
that referenced
this pull request
Feb 7, 2024
# [0.34.0](feast-dev/feast@v0.33.0...v0.34.0) (2023-09-07) ### Bug Fixes * Add NUMERIC to bq_to_feast type map ([feast-dev#3719](feast-dev#3719)) ([6474b4b](feast-dev@6474b4b)) * Fix python unit tests ([feast-dev#3734](feast-dev#3734)) ([e81684d](feast-dev@e81684d)) * Handle unknown postgres source types gracefully ([feast-dev#3634](feast-dev#3634)) ([d7041f4](feast-dev@d7041f4)) * Pin protobuf version to avoid seg fault on some machines ([028cc20](feast-dev@028cc20)) * Remove unwanted excessive splitting of gcs path, so expected gcs parquet paths are returned from BigQueryRetrievalJob.to_remote_storage() ([feast-dev#3730](feast-dev#3730)) ([f2c5988](feast-dev@f2c5988)) * Run store.plan() only when need it. ([feast-dev#3708](feast-dev#3708)) ([7bc7c47](feast-dev@7bc7c47)) * Saved datasets no longer break CLI registry-dump command ([feast-dev#3717](feast-dev#3717)) ([f28ccc2](feast-dev@f28ccc2)) * Update py3.8 ci requirements for cython 3.0 release ([feast-dev#3735](feast-dev#3735)) ([1695c13](feast-dev@1695c13)) ### Features * Enhance customization of Trino connections when using Trino-based Offline Stores ([feast-dev#3699](feast-dev#3699)) ([ed7535e](feast-dev@ed7535e)) * Implement gRPC server to ingest streaming features ([feast-dev#3687](feast-dev#3687)) ([a3fcd1f](feast-dev@a3fcd1f)) Signed-off-by: Attila Toth <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Right now when trying to read a postgres table, if there are any unknown types even if you're using a query on top of the table with only valid types it will throw an exception. Technically you can avoid this by creating a view on top of your table but this can be a hassle when the custom query is serving this purpose anyways imo
With this fix, the only change is to surface unknown types as "unknown", which your code handles as-is with a warning and ignoring the column, allowing for usage of these tables without needing to support the arbitrary types themselves.
Which issue(s) this PR fixes: