Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for ingesting from BigQuery SQL statement / View #6

Closed
pradithya opened this issue Dec 13, 2018 · 5 comments
Closed

Add support for ingesting from BigQuery SQL statement / View #6

pradithya opened this issue Dec 13, 2018 · 5 comments
Labels
kind/feature New feature or request

Comments

@pradithya
Copy link
Collaborator

Currently Feast only support ingesting feature from BigQuery Table.
@pradithya pradithya added the kind/feature New feature or request label Dec 13, 2018
@woop
Copy link
Member

woop commented Dec 14, 2018 •
edited
Loading

This issue isn't clearly described. Feast supports ingesting features from BigQuery, CSVs, as well as streams (Pub/Sub & Kafka).

@pradithya
Copy link
Collaborator Author

My bad. Yes, feast supports ingesting features from all those sources, although currently for BigQuery case it is still limited to consuming a table. Adding supports to ingest from BQ view or SQL query could add more flexibility of data source.

@bethrice44
Copy link

Any updates on this? It would be really useful to be able to ingest views

@woop
Copy link
Member

woop commented Jul 20, 2019

Any updates on this? It would be really useful to be able to ingest views

Hi Beth,

We are currently making some significant breaking changes to Feast, after which we expect to be able to ingest BQ views.

@woop
Copy link
Member

woop commented Jan 26, 2020

Closing this issue for now. It is based on Feast 0.1. We will be revisiting this with enhancements to the feature set specification.

@woop woop closed this as completed Jan 26, 2020
@woop woop mentioned this issue May 23, 2020
Closed
Yanson pushed a commit to Yanson/feast that referenced this issue Jul 29, 2020
@algattik
Run end-2-end tests in GitHub Actions CI (feast-dev#6)
e0158e3 
Uses docker-compose to deploy alongside with Feast (Core, Kafka, PostgreSQL) containers, a CI container that runs end-to-end tests.

As docker-compose runs the container in the same network, it can access the other containers directly through the docker network.

With this PR we run a single basic integration test for Feast Core only, to validate the concept. Later on we will try to also deploy the serving container, and run more end-to-end tests.
gitbook-com bot pushed a commit that referenced this issue Nov 25, 2021
@adchia @gitbook-bot
GitBook: [#6] Update 3P integrations with Snowflake for 0.15
4633037
KarolisKont referenced this issue in vinted/feast Oct 17, 2023
@james-crabtree-sp
SAASMLOPS-809 fix bytewax workers so they only process a single file (#6
49497ec 
)

* SAASMLOPS-809 fix bytewax workers so they only process a single file

* SAASMLOPS-809 fix newlines

Signed-off-by: James Crabtree <[email protected]>
achals pushed a commit that referenced this issue Oct 25, 2023
@james-crabtree-sp
fix: Redundant feature materialization and premature incremental mate…
417b16b 
…rialization timestamp updates (#3789)

* SAASMLOPS-767 wait for jobs to complete

Signed-off-by: James Crabtree <[email protected]>

* SAASMLOPS-805 Stopgap change to fix duplicate materialization of data

Signed-off-by: James Crabtree <[email protected]>

* SAASMLOPS-805 save BYTEWAX_REPLICAS=1

Signed-off-by: James Crabtree <[email protected]>

* SAASMLOPS-809 fix bytewax workers so they only process a single file (#6)

* SAASMLOPS-809 fix bytewax workers so they only process a single file

* SAASMLOPS-809 fix newlines

Signed-off-by: James Crabtree <[email protected]>

* SAASMLOPS-833 add configurable job timeout (#7)

* SAASMLOPS-833 add configurable job timeout

* SAASMLOPS-833 fix whitespace

Signed-off-by: James Crabtree <[email protected]>

* develop Run large materializations in batches of pods

Signed-off-by: James Crabtree <[email protected]>

* master Set job_batch_size at least equal to max_parallelism

Signed-off-by: James Crabtree <[email protected]>

* master clarity max_parallelism description

Signed-off-by: James Crabtree <[email protected]>

* master resolve bug that causes materialization to continue after job error

Signed-off-by: James Crabtree <[email protected]>

* master resolve bug causing pod logs to not be printed

Signed-off-by: James Crabtree <[email protected]>

---------

Signed-off-by: James Crabtree <[email protected]>
woop pushed a commit that referenced this issue Jan 13, 2024
@feast-ci-bot
chore(release): release 0.35.0
9237361 
# [0.35.0](v0.34.0...v0.35.0) (2024-01-13)

### Bug Fixes

* Add async refresh to prevent synchronous refresh in main thread ([#3812](#3812)) ([9583ed6](9583ed6))
* Adopt connection pooling for HBase ([#3793](#3793)) ([b3852bf](b3852bf))
* Bytewax engine create configmap from object ([#3821](#3821)) ([25e9775](25e9775))
* Fix warnings from deprecated paths and update default log level ([#3757](#3757)) ([68a8737](68a8737))
* improve parsing bytewax job status ([5983f40](5983f40))
* make bytewax settings unexposed ([ae1bb8b](ae1bb8b))
* Make generated temp table name escaped ([#3797](#3797)) ([175d796](175d796))
* Pin numpy version to avoid spammy deprecation messages ([774ed33](774ed33))
* Redundant feature materialization and premature incremental materialization timestamp updates ([#3789](#3789)) ([417b16b](417b16b)), closes [#6](#6) [#7](#7)
* Resolve hbase hotspot issue when materializing ([#3790](#3790)) ([7376db8](7376db8))
* Set keepalives_idle None by default ([#3756](#3756)) ([8717e9b](8717e9b))
* Set upper bound for bigquery client due to its breaking changes ([2151c39](2151c39))
* UI project cannot handle fallback routes ([#3766](#3766)) ([96ece0f](96ece0f))
* update dependencies versions due to conflicts ([5dc0b24](5dc0b24))
* Update jackson and remove unnecessary logging ([#3809](#3809)) ([018d0ea](018d0ea))
* upgrade the pyarrow to latest v14.0.1 for CVE-2023-47248. ([052182b](052182b))

### Features

* Add get online feature rpc to gprc server ([#3815](#3815)) ([01db8cc](01db8cc))
* Add materialize and materialize-incremental rest endpoints ([#3761](#3761)) ([fa600fe](fa600fe)), closes [#3760](#3760)
* add redis sentinel support ([3387a15](3387a15))
* add redis sentinel support ([4337c89](4337c89))
* add redis sentinel support format lint ([aad8718](aad8718))
* Add support for `table_create_disposition` in bigquery job for offline store ([#3762](#3762)) ([6a728fe](6a728fe))
* Add support for in_cluster config and additional labels for bytewax materialization ([#3754](#3754)) ([2192e65](2192e65))
* Apply cache to load proto registry for performance ([#3702](#3702)) ([709c709](709c709))
* Make bytewax job write as mini-batches ([#3777](#3777)) ([9b0e5ce](9b0e5ce))
* Optimize bytewax pod resource with zero-copy ([9cf9d96](9cf9d96))
* Support GCS filesystem for bytewax engine ([#3774](#3774)) ([fb6b807](fb6b807))
tokoko pushed a commit to tokoko/feast that referenced this issue Feb 6, 2024
@feast-ci-bot @tokoko
chore(release): release 0.35.0
85b5b55 
# [0.35.0](feast-dev/feast@v0.34.0...v0.35.0) (2024-01-13)

### Bug Fixes

* Add async refresh to prevent synchronous refresh in main thread ([feast-dev#3812](feast-dev#3812)) ([9583ed6](feast-dev@9583ed6))
* Adopt connection pooling for HBase ([feast-dev#3793](feast-dev#3793)) ([b3852bf](feast-dev@b3852bf))
* Bytewax engine create configmap from object ([feast-dev#3821](feast-dev#3821)) ([25e9775](feast-dev@25e9775))
* Fix warnings from deprecated paths and update default log level ([feast-dev#3757](feast-dev#3757)) ([68a8737](feast-dev@68a8737))
* improve parsing bytewax job status ([5983f40](feast-dev@5983f40))
* make bytewax settings unexposed ([ae1bb8b](feast-dev@ae1bb8b))
* Make generated temp table name escaped ([feast-dev#3797](feast-dev#3797)) ([175d796](feast-dev@175d796))
* Pin numpy version to avoid spammy deprecation messages ([774ed33](feast-dev@774ed33))
* Redundant feature materialization and premature incremental materialization timestamp updates ([feast-dev#3789](feast-dev#3789)) ([417b16b](feast-dev@417b16b)), closes [feast-dev#6](feast-dev#6) [feast-dev#7](feast-dev#7)
* Resolve hbase hotspot issue when materializing ([feast-dev#3790](feast-dev#3790)) ([7376db8](feast-dev@7376db8))
* Set keepalives_idle None by default ([feast-dev#3756](feast-dev#3756)) ([8717e9b](feast-dev@8717e9b))
* Set upper bound for bigquery client due to its breaking changes ([2151c39](feast-dev@2151c39))
* UI project cannot handle fallback routes ([feast-dev#3766](feast-dev#3766)) ([96ece0f](feast-dev@96ece0f))
* update dependencies versions due to conflicts ([5dc0b24](feast-dev@5dc0b24))
* Update jackson and remove unnecessary logging ([feast-dev#3809](feast-dev#3809)) ([018d0ea](feast-dev@018d0ea))
* upgrade the pyarrow to latest v14.0.1 for CVE-2023-47248. ([052182b](feast-dev@052182b))

### Features

* Add get online feature rpc to gprc server ([feast-dev#3815](feast-dev#3815)) ([01db8cc](feast-dev@01db8cc))
* Add materialize and materialize-incremental rest endpoints ([feast-dev#3761](feast-dev#3761)) ([fa600fe](feast-dev@fa600fe)), closes [feast-dev#3760](feast-dev#3760)
* add redis sentinel support ([3387a15](feast-dev@3387a15))
* add redis sentinel support ([4337c89](feast-dev@4337c89))
* add redis sentinel support format lint ([aad8718](feast-dev@aad8718))
* Add support for `table_create_disposition` in bigquery job for offline store ([feast-dev#3762](feast-dev#3762)) ([6a728fe](feast-dev@6a728fe))
* Add support for in_cluster config and additional labels for bytewax materialization ([feast-dev#3754](feast-dev#3754)) ([2192e65](feast-dev@2192e65))
* Apply cache to load proto registry for performance ([feast-dev#3702](feast-dev#3702)) ([709c709](feast-dev@709c709))
* Make bytewax job write as mini-batches ([feast-dev#3777](feast-dev#3777)) ([9b0e5ce](feast-dev@9b0e5ce))
* Optimize bytewax pod resource with zero-copy ([9cf9d96](feast-dev@9cf9d96))
* Support GCS filesystem for bytewax engine ([feast-dev#3774](feast-dev#3774)) ([fb6b807](feast-dev@fb6b807))

Signed-off-by: tokoko <[email protected]>
zseta pushed a commit to zseta/feast that referenced this issue Feb 7, 2024
@james-crabtree-sp @zseta
fix: Redundant feature materialization and premature incremental mate…
75dbaef 
…rialization timestamp updates (feast-dev#3789)

* SAASMLOPS-767 wait for jobs to complete

Signed-off-by: James Crabtree <[email protected]>

* SAASMLOPS-805 Stopgap change to fix duplicate materialization of data

Signed-off-by: James Crabtree <[email protected]>

* SAASMLOPS-805 save BYTEWAX_REPLICAS=1

Signed-off-by: James Crabtree <[email protected]>

* SAASMLOPS-809 fix bytewax workers so they only process a single file (feast-dev#6)

* SAASMLOPS-809 fix bytewax workers so they only process a single file

* SAASMLOPS-809 fix newlines

Signed-off-by: James Crabtree <[email protected]>

* SAASMLOPS-833 add configurable job timeout (feast-dev#7)

* SAASMLOPS-833 add configurable job timeout

* SAASMLOPS-833 fix whitespace

Signed-off-by: James Crabtree <[email protected]>

* develop Run large materializations in batches of pods

Signed-off-by: James Crabtree <[email protected]>

* master Set job_batch_size at least equal to max_parallelism

Signed-off-by: James Crabtree <[email protected]>

* master clarity max_parallelism description

Signed-off-by: James Crabtree <[email protected]>

* master resolve bug that causes materialization to continue after job error

Signed-off-by: James Crabtree <[email protected]>

* master resolve bug causing pod logs to not be printed

Signed-off-by: James Crabtree <[email protected]>

---------

Signed-off-by: James Crabtree <[email protected]>
Signed-off-by: Attila Toth <[email protected]>
zseta pushed a commit to zseta/feast that referenced this issue Feb 7, 2024
@feast-ci-bot @zseta
chore(release): release 0.35.0
ef5ff9e 
# [0.35.0](feast-dev/feast@v0.34.0...v0.35.0) (2024-01-13)

### Bug Fixes

* Add async refresh to prevent synchronous refresh in main thread ([feast-dev#3812](feast-dev#3812)) ([9583ed6](feast-dev@9583ed6))
* Adopt connection pooling for HBase ([feast-dev#3793](feast-dev#3793)) ([b3852bf](feast-dev@b3852bf))
* Bytewax engine create configmap from object ([feast-dev#3821](feast-dev#3821)) ([25e9775](feast-dev@25e9775))
* Fix warnings from deprecated paths and update default log level ([feast-dev#3757](feast-dev#3757)) ([68a8737](feast-dev@68a8737))
* improve parsing bytewax job status ([5983f40](feast-dev@5983f40))
* make bytewax settings unexposed ([ae1bb8b](feast-dev@ae1bb8b))
* Make generated temp table name escaped ([feast-dev#3797](feast-dev#3797)) ([175d796](feast-dev@175d796))
* Pin numpy version to avoid spammy deprecation messages ([774ed33](feast-dev@774ed33))
* Redundant feature materialization and premature incremental materialization timestamp updates ([feast-dev#3789](feast-dev#3789)) ([417b16b](feast-dev@417b16b)), closes [feast-dev#6](feast-dev#6) [feast-dev#7](feast-dev#7)
* Resolve hbase hotspot issue when materializing ([feast-dev#3790](feast-dev#3790)) ([7376db8](feast-dev@7376db8))
* Set keepalives_idle None by default ([feast-dev#3756](feast-dev#3756)) ([8717e9b](feast-dev@8717e9b))
* Set upper bound for bigquery client due to its breaking changes ([2151c39](feast-dev@2151c39))
* UI project cannot handle fallback routes ([feast-dev#3766](feast-dev#3766)) ([96ece0f](feast-dev@96ece0f))
* update dependencies versions due to conflicts ([5dc0b24](feast-dev@5dc0b24))
* Update jackson and remove unnecessary logging ([feast-dev#3809](feast-dev#3809)) ([018d0ea](feast-dev@018d0ea))
* upgrade the pyarrow to latest v14.0.1 for CVE-2023-47248. ([052182b](feast-dev@052182b))

### Features

* Add get online feature rpc to gprc server ([feast-dev#3815](feast-dev#3815)) ([01db8cc](feast-dev@01db8cc))
* Add materialize and materialize-incremental rest endpoints ([feast-dev#3761](feast-dev#3761)) ([fa600fe](feast-dev@fa600fe)), closes [feast-dev#3760](feast-dev#3760)
* add redis sentinel support ([3387a15](feast-dev@3387a15))
* add redis sentinel support ([4337c89](feast-dev@4337c89))
* add redis sentinel support format lint ([aad8718](feast-dev@aad8718))
* Add support for `table_create_disposition` in bigquery job for offline store ([feast-dev#3762](feast-dev#3762)) ([6a728fe](feast-dev@6a728fe))
* Add support for in_cluster config and additional labels for bytewax materialization ([feast-dev#3754](feast-dev#3754)) ([2192e65](feast-dev@2192e65))
* Apply cache to load proto registry for performance ([feast-dev#3702](feast-dev#3702)) ([709c709](feast-dev@709c709))
* Make bytewax job write as mini-batches ([feast-dev#3777](feast-dev#3777)) ([9b0e5ce](feast-dev@9b0e5ce))
* Optimize bytewax pod resource with zero-copy ([9cf9d96](feast-dev@9cf9d96))
* Support GCS filesystem for bytewax engine ([feast-dev#3774](feast-dev#3774)) ([fb6b807](feast-dev@fb6b807))

Signed-off-by: Attila Toth <[email protected]>
dmartinol added a commit to dmartinol/feast that referenced this issue May 17, 2024
@dmartinol
Merge pull request feast-dev#6 from redhatHameed/update_client
cb27b24 
added unit test for offline store remote client
tsisodia10 pushed a commit to tsisodia10/feast that referenced this issue Jul 23, 2024
@dependabot @tsisodia10
# This is a combination of 7 commits.
eb40d73 
# This is the 1st commit message:

chore: Bump ws from 7.5.6 to 7.5.10 in /ui (feast-dev#4292)


# This is the commit message feast-dev#2:

Remove metric checks

Signed-off-by: Twinkll Sisodia <[email protected]>

# This is the commit message feast-dev#3:

Draft different md file

# This is the commit message feast-dev#4:

Refine opentelemetry.md

# This is the commit message feast-dev#5:

Refine opentelemetry.md

# This is the commit message feast-dev#6:

Refine opentelemetry.md

# This is the commit message feast-dev#7:

Refine opentelemetry.md
tsisodia10 pushed a commit to tsisodia10/feast that referenced this issue Jul 23, 2024
@dependabot @tsisodia10
# This is a combination of 7 commits.
f036670 
# This is the 1st commit message:

chore: Bump ws from 7.5.6 to 7.5.10 in /ui (feast-dev#4292)


# This is the commit message feast-dev#2:

Remove metric checks

Signed-off-by: Twinkll Sisodia <[email protected]>

# This is the commit message feast-dev#3:

Draft different md file

# This is the commit message feast-dev#4:

Refine opentelemetry.md

# This is the commit message feast-dev#5:

Refine opentelemetry.md

# This is the commit message feast-dev#6:

Refine opentelemetry.md

# This is the commit message feast-dev#7:

Refine opentelemetry.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pradithya @woop @bethrice44