fix: Redshift push ignores schema #3671
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Robin Neufeld <[email protected]>
Signed-off-by: Robin Neufeld <[email protected]>
Signed-off-by: Robin Neufeld <[email protected]>
Signed-off-by: Robin Neufeld <[email protected]>
Signed-off-by: Robin Neufeld <[email protected]>
Signed-off-by: Robin Neufeld <[email protected]>
Signed-off-by: Robin Neufeld <[email protected]>
Signed-off-by: Robin Neufeld <[email protected]>
metavee
changed the title
|
Unit tests may be failing due to this issue between pip-tools and the latest version of pip Edit: might be fixed now? |
Signed-off-by: Robin Neufeld <[email protected]>
Signed-off-by: Robin Neufeld <[email protected]>
metavee
force-pushed
the
redshift-push-to-schema
branch
from
3bbfc06
to
6529b2a
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: adchia, metavee The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Manisha4
added a commit
to ExpediaGroup/feast
that referenced
this pull request
Aug 11, 2023
* ci: Add bigtable cleanup script Signed-off-by: Danny C <[email protected]> * fix: Missing Catalog argument in athena connector (feast-dev#3661) update Catalog argument in athena connector Signed-off-by: Gyumin Lee <[email protected]> Co-authored-by: Gyumin Lee <[email protected]> * ci: Disable flaky lambda materialization test Signed-off-by: Danny C <[email protected]> * fix: Broken non-root path with projects-list.json (feast-dev#3665) ensure correct precedence with the two operators Signed-off-by: Ben Fletcher <[email protected]> * fix: Manage redis pipe's context (feast-dev#3655) Signed-off-by: Jiwon Park <[email protected]> * chore: Bump tough-cookie from 4.0.0 to 4.1.3 in /sdk/python/feast/ui (feast-dev#3677) Bumps [tough-cookie](https://github.com/salesforce/tough-cookie) from 4.0.0 to 4.1.3. - [Release notes](https://github.com/salesforce/tough-cookie/releases) - [Changelog](https://github.com/salesforce/tough-cookie/blob/master/CHANGELOG.md) - [Commits](salesforce/tough-cookie@v4.0.0...v4.1.3) --- updated-dependencies: - dependency-name: tough-cookie dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: Bump tough-cookie from 4.0.0 to 4.1.3 in /ui (feast-dev#3676) Bumps [tough-cookie](https://github.com/salesforce/tough-cookie) from 4.0.0 to 4.1.3. - [Release notes](https://github.com/salesforce/tough-cookie/releases) - [Changelog](https://github.com/salesforce/tough-cookie/blob/master/CHANGELOG.md) - [Commits](salesforce/tough-cookie@v4.0.0...v4.1.3) --- updated-dependencies: - dependency-name: tough-cookie dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: For SQL registry, increase max data_source_name length to 255 (feast-dev#3630) * sql.py data_sources.data_source_name String(255) Extend the limit of the data_source_name field from 50 to 255. Signed-off-by: Ross Donnachie <[email protected]> * fix: Optimize bytes processed when retrieving entity df schema to 0 (feast-dev#3680) feat: Optimize bytes processed when retrieving entity df schema to 0 Signed-off-by: Hai Nguyen <[email protected]> * fix: Entityless fv breaks with `KeyError: __dummy` applying feature_store.plan() on python (feast-dev#3640) * fix! KeyError: __dummy on entityless fv Signed-off-by: williamfoschiera <[email protected]> * fix! join_keys typing. Signed-off-by: williamfoschiera <[email protected]> --------- Signed-off-by: williamfoschiera <[email protected]> Co-authored-by: williamfoschiera <[email protected]> * chore: Bump protobufjs from 7.1.1 to 7.2.4 in /ui (feast-dev#3674) Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.1.1 to 7.2.4. - [Release notes](https://github.com/protobufjs/protobuf.js/releases) - [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md) - [Commits](protobufjs/protobuf.js@protobufjs-v7.1.1...protobufjs-v7.2.4) --- updated-dependencies: - dependency-name: protobufjs dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: Bump protobufjs from 7.1.2 to 7.2.4 in /sdk/python/feast/ui (feast-dev#3675) Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.1.2 to 7.2.4. - [Release notes](https://github.com/protobufjs/protobuf.js/releases) - [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md) - [Commits](protobufjs/protobuf.js@protobufjs-v7.1.2...protobufjs-v7.2.4) --- updated-dependencies: - dependency-name: protobufjs dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: Bump semver from 6.3.0 to 6.3.1 in /ui (feast-dev#3678) Bumps [semver](https://github.com/npm/node-semver) from 6.3.0 to 6.3.1. - [Release notes](https://github.com/npm/node-semver/releases) - [Changelog](https://github.com/npm/node-semver/blob/v6.3.1/CHANGELOG.md) - [Commits](npm/node-semver@v6.3.0...v6.3.1) --- updated-dependencies: - dependency-name: semver dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: Bump semver from 6.3.0 to 6.3.1 in /sdk/python/feast/ui (feast-dev#3679) Bumps [semver](https://github.com/npm/node-semver) from 6.3.0 to 6.3.1. - [Release notes](https://github.com/npm/node-semver/releases) - [Changelog](https://github.com/npm/node-semver/blob/v6.3.1/CHANGELOG.md) - [Commits](npm/node-semver@v6.3.0...v6.3.1) --- updated-dependencies: - dependency-name: semver dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: Bump google.golang.org/grpc from 1.47.0 to 1.53.0 (feast-dev#3670) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.47.0 to 1.53.0. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.47.0...v1.53.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(release): release 0.32.0 # [0.32.0](feast-dev/feast@v0.31.0...v0.32.0) (2023-07-17) ### Bug Fixes * Added generic Feature store Creation for CLI ([feast-dev#3618](feast-dev#3618)) ([bf740d2](feast-dev@bf740d2)) * Broken non-root path with projects-list.json ([feast-dev#3665](feast-dev#3665)) ([4861af0](feast-dev@4861af0)) * Clean up snowflake to_spark_df() ([feast-dev#3607](feast-dev#3607)) ([e8e643e](feast-dev@e8e643e)) * Entityless fv breaks with `KeyError: __dummy` applying feature_store.plan() on python ([feast-dev#3640](feast-dev#3640)) ([ef4ef32](feast-dev@ef4ef32)) * Fix scan datasize to 0 for inference schema ([feast-dev#3628](feast-dev#3628)) ([c3dd74e](feast-dev@c3dd74e)) * Fix timestamp consistency in push api ([feast-dev#3614](feast-dev#3614)) ([9b227d7](feast-dev@9b227d7)) * For SQL registry, increase max data_source_name length to 255 ([feast-dev#3630](feast-dev#3630)) ([478caec](feast-dev@478caec)) * Implements connection pool for postgres online store ([feast-dev#3633](feast-dev#3633)) ([059509a](feast-dev@059509a)) * Manage redis pipe's context ([feast-dev#3655](feast-dev#3655)) ([48e0971](feast-dev@48e0971)) * Missing Catalog argument in athena connector ([feast-dev#3661](feast-dev#3661)) ([f6d3caf](feast-dev@f6d3caf)) * Optimize bytes processed when retrieving entity df schema to 0 ([feast-dev#3680](feast-dev#3680)) ([1c01035](feast-dev@1c01035)) ### Features * Add gunicorn for serve with multiprocess ([feast-dev#3636](feast-dev#3636)) ([4de7faf](feast-dev@4de7faf)) * Use string as a substitute for unregistered types during schema inference ([feast-dev#3646](feast-dev#3646)) ([c474ccd](feast-dev@c474ccd)) * fix: Redshift push ignores schema (feast-dev#3671) * Add fully-qualified-table-name Redshift prop Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * Docstring Signed-off-by: Robin Neufeld <[email protected]> * Test fully_qualified_table_name Signed-off-by: Robin Neufeld <[email protected]> * Simplify logic Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * Test offline_write_batch Signed-off-by: Robin Neufeld <[email protected]> * Bump to trigger CI Signed-off-by: Robin Neufeld <[email protected]> * another bump for ci Signed-off-by: Robin Neufeld <[email protected]> --------- Signed-off-by: Robin Neufeld <[email protected]> * fix: Add aws-sts dependency in java sdk so that S3 client acquires IRSA role (feast-dev#3696) Add aws-sts dependency in java sdk Signed-off-by: harmeet-singh-discovery <[email protected]> * Adding initial update changes * Added formatting changes * Revert "Merge branch 'feast-dev:master' into msudhir/add-vector-update-functionality" This reverts commit 8487678, reversing changes made to 0578b9b. * Added more tests and functionality * updating tests * updated functionality and added more tests * correcting a test case * Making formatting corrections and changeing log * Improved tests and added functionality to convert feast schema to milvus readable schema * Added PR Review comments * Fixed failing test --------- Signed-off-by: Danny C <[email protected]> Signed-off-by: Gyumin Lee <[email protected]> Signed-off-by: Ben Fletcher <[email protected]> Signed-off-by: Jiwon Park <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Ross Donnachie <[email protected]> Signed-off-by: Hai Nguyen <[email protected]> Signed-off-by: williamfoschiera <[email protected]> Signed-off-by: Robin Neufeld <[email protected]> Signed-off-by: harmeet-singh-discovery <[email protected]> Co-authored-by: Danny C <[email protected]> Co-authored-by: 이규민 <[email protected]> Co-authored-by: Gyumin Lee <[email protected]> Co-authored-by: Ben Fletcher <[email protected]> Co-authored-by: Jiwon Park <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ross Donnachie <[email protected]> Co-authored-by: Harry <[email protected]> Co-authored-by: William Foschiera <[email protected]> Co-authored-by: williamfoschiera <[email protected]> Co-authored-by: feast-ci-bot <[email protected]> Co-authored-by: Robin Neufeld <[email protected]> Co-authored-by: harmeet-singh-discovery <[email protected]> Co-authored-by: Manisha Sudhir <[email protected]>
felixwang9817
pushed a commit
that referenced
this pull request
Aug 14, 2023
felixwang9817
pushed a commit
that referenced
this pull request
Aug 14, 2023
# [0.33.0](v0.32.0...v0.33.0) (2023-08-14) ### Bug Fixes * Add aws-sts dependency in java sdk so that S3 client acquires IRSA role ([#3696](#3696)) ([c75a01f](c75a01f)) * Redshift push ignores schema ([#3671](#3671)) ([76270f6](76270f6)) ### Features * Add possibility to save dataset as table, when spark config has remote warehouse info ([#3645](#3645)) ([22c109b](22c109b))
zseta
pushed a commit
to zseta/feast
that referenced
this pull request
Feb 7, 2024
* Add fully-qualified-table-name Redshift prop Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * Docstring Signed-off-by: Robin Neufeld <[email protected]> * Test fully_qualified_table_name Signed-off-by: Robin Neufeld <[email protected]> * Simplify logic Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * Test offline_write_batch Signed-off-by: Robin Neufeld <[email protected]> * Bump to trigger CI Signed-off-by: Robin Neufeld <[email protected]> * another bump for ci Signed-off-by: Robin Neufeld <[email protected]> --------- Signed-off-by: Robin Neufeld <[email protected]> Signed-off-by: Attila Toth <[email protected]>
zseta
pushed a commit
to zseta/feast
that referenced
this pull request
Feb 7, 2024
# [0.33.0](feast-dev/feast@v0.32.0...v0.33.0) (2023-08-14) ### Bug Fixes * Add aws-sts dependency in java sdk so that S3 client acquires IRSA role ([feast-dev#3696](feast-dev#3696)) ([c75a01f](feast-dev@c75a01f)) * Redshift push ignores schema ([feast-dev#3671](feast-dev#3671)) ([76270f6](feast-dev@76270f6)) ### Features * Add possibility to save dataset as table, when spark config has remote warehouse info ([feast-dev#3645](feast-dev#3645)) ([22c109b](feast-dev@22c109b)) Signed-off-by: Attila Toth <[email protected]>
zerafachris
pushed a commit
to zerafachris/feast
that referenced
this pull request
Mar 5, 2024
* Add fully-qualified-table-name Redshift prop Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * Docstring Signed-off-by: Robin Neufeld <[email protected]> * Test fully_qualified_table_name Signed-off-by: Robin Neufeld <[email protected]> * Simplify logic Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * Test offline_write_batch Signed-off-by: Robin Neufeld <[email protected]> * Bump to trigger CI Signed-off-by: Robin Neufeld <[email protected]> * another bump for ci Signed-off-by: Robin Neufeld <[email protected]> --------- Signed-off-by: Robin Neufeld <[email protected]> Signed-off-by: zerafachris PERSONAL <[email protected]>
EXPEbdodla
pushed a commit
to ExpediaGroup/feast
that referenced
this pull request
Jun 10, 2024
* Add fully-qualified-table-name Redshift prop Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * Docstring Signed-off-by: Robin Neufeld <[email protected]> * Test fully_qualified_table_name Signed-off-by: Robin Neufeld <[email protected]> * Simplify logic Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * Test offline_write_batch Signed-off-by: Robin Neufeld <[email protected]> * Bump to trigger CI Signed-off-by: Robin Neufeld <[email protected]> * another bump for ci Signed-off-by: Robin Neufeld <[email protected]> --------- Signed-off-by: Robin Neufeld <[email protected]>
EXPEbdodla
pushed a commit
to ExpediaGroup/feast
that referenced
this pull request
Jun 10, 2024
# [0.33.0](feast-dev/feast@v0.32.0...v0.33.0) (2023-08-14) ### Bug Fixes * Add aws-sts dependency in java sdk so that S3 client acquires IRSA role ([feast-dev#3696](feast-dev#3696)) ([c75a01f](feast-dev@c75a01f)) * Redshift push ignores schema ([feast-dev#3671](feast-dev#3671)) ([76270f6](feast-dev@76270f6)) ### Features * Add possibility to save dataset as table, when spark config has remote warehouse info ([feast-dev#3645](feast-dev#3645)) ([22c109b](feast-dev@22c109b))
EXPEbdodla
pushed a commit
to ExpediaGroup/feast
that referenced
this pull request
Jun 11, 2024
* Add fully-qualified-table-name Redshift prop Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * Docstring Signed-off-by: Robin Neufeld <[email protected]> * Test fully_qualified_table_name Signed-off-by: Robin Neufeld <[email protected]> * Simplify logic Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * pre-commit Signed-off-by: Robin Neufeld <[email protected]> * Test offline_write_batch Signed-off-by: Robin Neufeld <[email protected]> * Bump to trigger CI Signed-off-by: Robin Neufeld <[email protected]> * another bump for ci Signed-off-by: Robin Neufeld <[email protected]> --------- Signed-off-by: Robin Neufeld <[email protected]>
EXPEbdodla
pushed a commit
to ExpediaGroup/feast
that referenced
this pull request
Jun 11, 2024
# [0.33.0](feast-dev/feast@v0.32.0...v0.33.0) (2023-08-14) ### Bug Fixes * Add aws-sts dependency in java sdk so that S3 client acquires IRSA role ([feast-dev#3696](feast-dev#3696)) ([c75a01f](feast-dev@c75a01f)) * Redshift push ignores schema ([feast-dev#3671](feast-dev#3671)) ([76270f6](feast-dev@76270f6)) ### Features * Add possibility to save dataset as table, when spark config has remote warehouse info ([feast-dev#3645](feast-dev#3645)) ([22c109b](feast-dev@22c109b))
EXPEbdodla
added a commit
to ExpediaGroup/feast
that referenced
this pull request
Jun 11, 2024
This reverts commit 2810d97.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
When Feast writes to a Redshift offline store, this PR ensures that it takes the fully qualified table name into consideration so that it does not write to the wrong place in the database.
Which issue(s) this PR fixes:
Fixes #3651