Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix duplicate upload entity #1981

Merged
merged 1 commit into from
Oct 27, 2021
Merged

Fix duplicate upload entity #1981

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 19 additions & 31 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,23 +129,18 @@ pprint(feature_vector)

The list below contains the functionality that contributors are planning to develop for Feast

* Items below that are in development (or planned for development) will be indicated in parentheses.
* Items below that are in development (or planned for development) will be indicated in parentheses.
* We welcome contribution to all items in the roadmap!
* Want to influence our roadmap and prioritization? Submit your feedback to [this form](https://docs.google.com/forms/d/e/1FAIpQLSfa1nRQ0sKz-JEFnMMCi4Jseag_yDssO\_3nV9qMfxfrkil-wA/viewform).
* Want to influence our roadmap and prioritization? Submit your feedback to [this form](https://docs.google.com/forms/d/e/1FAIpQLSfa1nRQ0sKz-JEFnMMCi4Jseag\_yDssO\_3nV9qMfxfrkil-wA/viewform).
* Want to speak to a Feast contributor? We are more than happy to jump on a call. Please schedule a time using [Calendly](https://calendly.com/d/x2ry-g5bb/meet-with-feast-team).



* **Data Sources**
* [x] [Redshift source](https://docs.feast.dev/reference/data-sources/redshift)
* [x] [BigQuery source](https://docs.feast.dev/reference/data-sources/bigquery)
* [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file)
* [ ] Kafka source (Planned for Q4 2021)
* [ ] Synapse source (Planned for Q4 2021)
* [ ] Snowflake source (Planned for Q4 2021)
* [ ] HTTP source


* **Data Sources**
* [x] [Redshift source](https://docs.feast.dev/reference/data-sources/redshift)
* [x] [BigQuery source](https://docs.feast.dev/reference/data-sources/bigquery)
* [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file)
* [ ] Kafka source (Planned for Q4 2021)
* [ ] Synapse source (Planned for Q4 2021)
* [ ] Snowflake source (Planned for Q4 2021)
* [ ] HTTP source
* **Offline Stores**
* [x] [Redshift](https://docs.feast.dev/reference/offline-stores/redshift)
* [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
Expand All @@ -154,8 +149,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Hive (community maintained)](https://github.com/baineng/feast-hive)
* [x] [Postgres (community maintained)](https://github.com/nossrannug/feast-postgres)
* [ ] Snowflake (Planned for Q4 2021)
* [ ] Synapse (Planned for Q4 2021)\

* [ ] Synapse (Planned for Q4 2021)
* **Online Stores**
* [x] [DynamoDB](https://docs.feast.dev/reference/online-stores/dynamodb)
* [x] [Redis](https://docs.feast.dev/reference/online-stores/redis)
Expand All @@ -164,41 +158,35 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
* [x] [Postgres (community maintained)](https://github.com/nossrannug/feast-postgres)
* [ ] Bigtable
* [ ] Cassandra\

* [ ] Cassandra\\
* **Streaming**
* [x] [Custom streaming ingestion job support](https://docs.feast.dev/how-to-guides/creating-a-custom-provider)
* [ ] Streaming ingestion on AWS (Planned for Q4 2021)
* [ ] Streaming ingestion on GCP\

* [ ] Streaming ingestion on GCP
* **Feature Engineering**
* [x] On-demand Transformations (Alpha release. See [RFC](https://docs.google.com/document/d/1lgfIw0Drc65LpaxbUu49RCeJgMew547meSJttnUqz7c/edit#))
* [ ] Batch transformation (SQL)
* [ ] Streaming transformation\

* [ ] Streaming transformation
* **Deployments**
* [x] AWS Lambda (Alpha release. See [RFC](https://docs.google.com/document/d/1eZWKWzfBif66LDN32IajpaG-j82LSHCCOzY6R7Ax7MI/edit))
* [ ] Cloud Run
* [ ] Kubernetes
* [ ] KNative\

* [ ] KNative\\
* **Feature Serving**
* [x] Python Client
* [x] REST Feature Server (Python) (Alpha release. See [RFC](https://docs.google.com/document/d/1iXvFhAsJ5jgAhPOpTdB3j-Wj1S9x3Ev_Wr6ZpnLzER4/edit))
* [x] REST Feature Server (Python) (Alpha release. See [RFC](https://docs.google.com/document/d/1iXvFhAsJ5jgAhPOpTdB3j-Wj1S9x3Ev\_Wr6ZpnLzER4/edit))
* [x] gRPC Feature Server (Java) (See [#1497](https://github.com/feast-dev/feast/issues/1497))
* [ ] Java Client
* [ ] Go Client
* [ ] Go Client
* [ ] Push API
* [ ] Delete API
* [ ] Feature Logging (for training)\

* [ ] Feature Logging (for training)
* **Data Quality Management**
* [ ] Data profiling and validation (Great Expectations) (Planned for Q4 2021)
* [ ] Metric production
* [ ] Training-serving skew detection
* [ ] Drift detection
* [ ] Alerting\

* [ ] Alerting
* **Feature Discovery and Governance**
* [x] Python SDK for browsing feature registry
* [x] CLI for browsing feature registry
Expand Down
29 changes: 15 additions & 14 deletions sdk/python/feast/infra/offline_stores/bigquery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,21 +249,22 @@ def to_bigquery(
Returns:
Returns the destination table name or returns None if job_config.dry_run is True.
"""
with self._query_generator() as query:
if not job_config:
today = date.today().strftime("%Y%m%d")
rand_id = str(uuid.uuid4())[:7]
path = f"{self.client.project}.{self.config.offline_store.dataset}.historical_{today}_{rand_id}"
job_config = bigquery.QueryJobConfig(destination=path)

if not job_config.dry_run and self.on_demand_feature_views:
job = self.client.load_table_from_dataframe(
self.to_df(), job_config.destination
)
job.result()
print(f"Done writing to '{job_config.destination}'.")
return str(job_config.destination)

if not job_config:
today = date.today().strftime("%Y%m%d")
rand_id = str(uuid.uuid4())[:7]
path = f"{self.client.project}.{self.config.offline_store.dataset}.historical_{today}_{rand_id}"
job_config = bigquery.QueryJobConfig(destination=path)

if not job_config.dry_run and self.on_demand_feature_views:
job = self.client.load_table_from_dataframe(
self.to_df(), job_config.destination
)
job.result()
print(f"Done writing to '{job_config.destination}'.")
return str(job_config.destination)

with self._query_generator() as query:
bq_job = self.client.query(query, job_config=job_config)

if job_config.dry_run:
Expand Down