diff --git a/README.md b/README.md index 64f6c5940f..efe194687f 100644 --- a/README.md +++ b/README.md @@ -139,7 +139,7 @@ The list below contains the functionality that contributors are planning to deve * Items below that are in development (or planned for development) will be indicated in parentheses. * We welcome contribution to all items in the roadmap! -* Want to speak to a Feast contributor? We are more than happy to jump on a call. Please schedule a time using [Calendly](https://calendly.com/d/x2ry-g5bb/meet-with-feast-team). +* Have questions about the roadmap? Go to the Slack channel to ask on #feast-development * **Data Sources** * [x] [Snowflake source](https://docs.feast.dev/reference/data-sources/snowflake) @@ -151,7 +151,6 @@ The list below contains the functionality that contributors are planning to deve * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres) * [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark) * [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push)) - * [ ] HTTP source * **Offline Stores** * [x] [Snowflake](https://docs.feast.dev/reference/offline-stores/snowflake) * [x] [Redshift](https://docs.feast.dev/reference/offline-stores/redshift) @@ -184,30 +183,20 @@ The list below contains the functionality that contributors are planning to deve * **Deployments** * [x] AWS Lambda (Alpha release. See [RFC](https://docs.google.com/document/d/1eZWKWzfBif66LDN32IajpaG-j82LSHCCOzY6R7Ax7MI/edit)) * [x] Kubernetes (See [guide](https://docs.feast.dev/how-to-guides/running-feast-in-production#4.3.-java-based-feature-server-deployed-on-kubernetes)) - * [ ] Cloud Run - * [ ] KNative * **Feature Serving** * [x] Python Client - * [x] REST Feature Server (Python) (Alpha release. See [RFC](https://docs.google.com/document/d/1iXvFhAsJ5jgAhPOpTdB3j-Wj1S9x3Ev\_Wr6ZpnLzER4/edit)) - * [x] gRPC Feature Server (Java) (See [#1497](https://github.com/feast-dev/feast/issues/1497)) - * [x] Push API - * [ ] Java Client - * [ ] Go Client - * [ ] Delete API - * [ ] Feature Logging (for training) + * [x] REST Feature Server (Python) (See [RFC](https://docs.google.com/document/d/1iXvFhAsJ5jgAhPOpTdB3j-Wj1S9x3Ev\_Wr6ZpnLzER4/edit)) + * [x] REST / gRPC Feature Server (Go) (Alpha release. See [docs](https://docs.feast.dev/reference/feature-servers/go-feature-retrieval) + * [x] gRPC Feature Server (Java) (Alpha release. See [#1497](https://github.com/feast-dev/feast/issues/1497)) * **Data Quality Management (See [RFC](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit))** * [x] Data profiling and validation (Great Expectations) - * [ ] Training-serving skew detection (in progress) - * [ ] Metric production - * [ ] Drift detection * **Feature Discovery and Governance** * [x] Python SDK for browsing feature registry * [x] CLI for browsing feature registry * [x] Model-centric feature tracking (feature services) * [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py)) + * [x] DataHub integration (see [DataHub Feast docs](https://datahubproject.io/docs/generated/ingestion/sources/feast/)) * [x] Feast Web UI (Alpha release. See [documentation](https://docs.feast.dev/reference/alpha-web-ui)) - * [ ] REST API for browsing feature registry - ## 🎓 Important Resources diff --git a/docs/community.md b/docs/community.md index c0ead3dda1..dc1cc8a0fe 100644 --- a/docs/community.md +++ b/docs/community.md @@ -1,16 +1,11 @@ # Community -{% hint style="success" %} -**Speak to us:** Have a question, feature request, idea, or just looking to speak to a real person? Set up a meeting with a Feast maintainer over [here](https://calendly.com/d/x2ry-g5bb/meet-with-feast-team)! -{% endhint %} - ## Links & Resources * [Slack](https://slack.feast.dev): Feel free to ask questions or say hello! * [Mailing list](https://groups.google.com/d/forum/feast-dev): We have both a user and developer mailing list. * Feast users should join [feast-discuss@googlegroups.com](mailto:feast-discuss@googlegroups.com) group by clicking [here](https://groups.google.com/g/feast-discuss). * Feast developers should join [feast-dev@googlegroups.com](mailto:feast-dev@googlegroups.com) group by clicking [here](https://groups.google.com/d/forum/feast-dev). - * People interested in the Feast community newsletter should join feast-announce by clicking [here](https://groups.google.com/d/forum/feast-announce). * [Community Calendar](https://calendar.google.com/calendar/u/0?cid=ZTFsZHVhdGM3MDU3YTJucTBwMzNqNW5rajBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ): Includes community calls and design meetings. * [Google Folder](https://drive.google.com/drive/u/0/folders/1jgMHOPDT2DvBlJeO9LCM79DP4lm4eOrR): This folder is used as a central repository for all Feast resources. For example: * Design proposals in the form of Request for Comments (RFC). @@ -27,7 +22,7 @@ ## Community Calls -We have a user and contributor community call every two weeks (Asia & US friendly). +We have a user and contributor community call every two weeks (US & EU friendly). {% hint style="info" %} Please join the above Feast user groups in order to see calendar invites to the community calls diff --git a/docs/getting-started/feast-workshop.md b/docs/getting-started/feast-workshop.md index c883625dac..8b6778c2d3 100644 --- a/docs/getting-started/feast-workshop.md +++ b/docs/getting-started/feast-workshop.md @@ -30,13 +30,15 @@ _See also:_ [_Feast quickstart_](https://docs.feast.dev/getting-started/quicksta These are meant mostly to be done in order, with examples building on previous concepts. -| Time (min) | Description | Module | -| :--------: | ----------------------------------------------------------------------- | --------------------------------------------------------------------------- | -| 30-45 | Setting up Feast projects & CI/CD + powering batch predictions | [Module 0](https://github.com/feast-dev/feast-workshop/tree/main/module\_0) | -| 15-20 | Streaming ingestion & online feature retrieval with Kafka, Spark, Redis | [Module 1](https://github.com/feast-dev/feast-workshop/tree/main/module\_1) | -| 10-15 | Real-time feature engineering with on demand transformations | [Module 2](https://github.com/feast-dev/feast-workshop/tree/main/module\_2) | -| TBD | Feature server deployment (embed, as a service, AWS Lambda) | TBD | -| TBD | Versioning features / models in Feast | TBD | -| TBD | Data quality monitoring in Feast | TBD | -| TBD | Batch transformations | TBD | -| TBD | Stream transformations | TBD | +See https://github.com/feast-dev/feast-workshop + +| Time (min) | Description | Module | +| :--------: | ----------------------------------------------------------------------- |-----------| +| 30-45 | Setting up Feast projects & CI/CD + powering batch predictions | Module 0 | +| 15-20 | Streaming ingestion & online feature retrieval with Kafka, Spark, Redis | Module 1 | +| 10-15 | Real-time feature engineering with on demand transformations | Module 2 | +| TBD | Feature server deployment (embed, as a service, AWS Lambda) | TBD | +| TBD | Versioning features / models in Feast | TBD | +| TBD | Data quality monitoring in Feast | TBD | +| TBD | Batch transformations | TBD | +| TBD | Stream transformations | TBD | diff --git a/docs/getting-started/third-party-integrations.md b/docs/getting-started/third-party-integrations.md index 8a862891f8..ef47a11029 100644 --- a/docs/getting-started/third-party-integrations.md +++ b/docs/getting-started/third-party-integrations.md @@ -11,55 +11,11 @@ Don't see your offline store or online store of choice here? Check out our guide ## Integrations -### **Data Sources** - -* [x] [Snowflake source](https://docs.feast.dev/reference/data-sources/snowflake) -* [x] [Redshift source](https://docs.feast.dev/reference/data-sources/redshift) -* [x] [BigQuery source](https://docs.feast.dev/reference/data-sources/bigquery) -* [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file) -* [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure) -* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive) -* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres) -* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark) -* [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push)) -* [ ] HTTP source - -### Offline Stores - -* [x] [Snowflake](https://docs.feast.dev/reference/offline-stores/snowflake) -* [x] [Redshift](https://docs.feast.dev/reference/offline-stores/redshift) -* [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery) -* [x] [Synapse (community plugin)](https://github.com/Azure/feast-azure) -* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive) -* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/offline-stores/postgres) -* [x] [Trino (contrib plugin)](https://github.com/Shopify/feast-trino) -* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/offline-stores/spark) -* [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file) -* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store) - -### Online Stores - -* [x] [DynamoDB](https://docs.feast.dev/reference/online-stores/dynamodb) -* [x] [Redis](https://docs.feast.dev/reference/online-stores/redis) -* [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore) -* [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite) -* [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure) -* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres) -* [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store) -* [x] [Cassandra / AstraDB](https://github.com/datastaxdevs/feast-cassandra-online-store) -* [ ] Bigtable (in progress) - -### **Deployments** - -* [x] AWS Lambda (Alpha release. See [guide](../reference/alpha-aws-lambda-feature-server.md) and [RFC](https://docs.google.com/document/d/1eZWKWzfBif66LDN32IajpaG-j82LSHCCOzY6R7Ax7MI/edit)) -* [x] Kubernetes (See [guide](https://docs.feast.dev/how-to-guides/running-feast-in-production#4.3.-java-based-feature-server-deployed-on-kubernetes)) -* [ ] Cloud Run -* [ ] KNative - +See [Functionality and Roadmap](../../README.md#-functionality-and-roadmap) ## Standards -In order for a plugin integration to be highlighted on this page, it must meet the following requirements: +In order for a plugin integration to be highlighted, it must meet the following requirements: 1. The plugin must have tests. Ideally it would use the Feast universal tests (see this [guide](../how-to-guides/adding-or-reusing-tests.md) for an example), but custom tests are fine. 2. The plugin must have some basic documentation on how it should be used. diff --git a/docs/project/development-guide.md b/docs/project/development-guide.md index 6d5bee16af..58e29a5ca7 100644 --- a/docs/project/development-guide.md +++ b/docs/project/development-guide.md @@ -11,85 +11,6 @@ This guide is targeted at developers looking to contribute to Feast: > Learn How the Feast [Contributing Process](contributing.md) works. -## Project Structure - -Feast is composed of [multiple components](../getting-started/architecture-and-components/) distributed into multiple repositories: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RepositoryDescriptionComponent(s)
Main Feast Repository - Hosts all required code to run Feast. This includes the Feast Python SDK - and Protobuf definitions. For legacy reasons this repository still contains - Terraform config and a Go Client for Feast. -
    -
  • Python SDK / CLI -
    • -
  • Protobuf APIs -
    • -
  • Documentation -
    • -
  • Go Client -
    • -
  • Terraform -
    • -
-
Feast Java - Java-specific Feast components. Includes the Feast Core Registry, Feast - Serving for serving online feature values, and the Feast Java Client for - retrieving feature values. -
    -
  • Core -
    • -
  • Serving -
    • -
  • Java Client -
    • -
-
Feast Spark - Feast Spark SDK & Feast Job Service for launching ingestion jobs and - for building training datasets with Spark -
    -
  • Spark SDK -
    • -
  • Job Service -
    • -
-
Feast Helm Chart - Helm Chart for deploying Feast on Kubernetes & Spark. -
    -
  • Helm Chart -
    • -
-
- ## Making a Pull Request {% hint style="info" %} @@ -148,5 +69,3 @@ The language specific bindings have to be regenerated when changes are made to t | :--- | :--- | :--- | | [Main Feast Repository](https://github.com/feast-dev/feast) | Python | Run `make compile-protos-python` to generate bindings | | [Main Feast Repository](https://github.com/feast-dev/feast) | Golang | Run `make compile-protos-go` to generate bindings | -| [Feast Java](https://github.com/feast-dev/feast-java) | Java | No action required: bindings are generated automatically during compilation. | - diff --git a/docs/project/new_branch_part_1.png b/docs/project/new_branch_part_1.png new file mode 100644 index 0000000000..e8e59d8214 Binary files /dev/null and b/docs/project/new_branch_part_1.png differ diff --git a/docs/project/new_branch_part_2.png b/docs/project/new_branch_part_2.png new file mode 100644 index 0000000000..f94c2e3227 Binary files /dev/null and b/docs/project/new_branch_part_2.png differ diff --git a/docs/project/new_branch_part_3.png b/docs/project/new_branch_part_3.png new file mode 100644 index 0000000000..34cbb80751 Binary files /dev/null and b/docs/project/new_branch_part_3.png differ diff --git a/docs/project/new_branch_part_4.png b/docs/project/new_branch_part_4.png new file mode 100644 index 0000000000..c9c3cc4352 Binary files /dev/null and b/docs/project/new_branch_part_4.png differ diff --git a/docs/project/new_branch_part_5.png b/docs/project/new_branch_part_5.png new file mode 100644 index 0000000000..89b3a08cc6 Binary files /dev/null and b/docs/project/new_branch_part_5.png differ diff --git a/docs/project/release-process.md b/docs/project/release-process.md index f7102d13c3..e9f3295d91 100644 --- a/docs/project/release-process.md +++ b/docs/project/release-process.md @@ -28,3 +28,30 @@ For Feast maintainers, these are the concrete steps for making a new release. 4. Try the dry run first with your personal access token. If this succeeds, uncheck `Dry Run` and run the release workflow. 5. All of the jobs should succeed besides the UI job which needs to be released separately. Ping a maintainer on Slack to run the UI release manually. 6. Try to install the feast release in your local environment and test out the `feast init` -> `feast apply` workflow to verify as a sanity check that the release worked correctly. + +### (for minor releases) Post-release steps +1. Create a new branch based on master (i.e. v0.22-branch) and push to the main Feast repo. This will be where cherry-picks go for future patch releases and where documentation will point. +2. Write a summary of the release in the GitHub release + 1. By default, Semantic Release will pull in messages from commits (features vs fixes, etc). But this is hard to digest still, so it helps to have a high level overview. + +### Update documentation + +In the Feast Gitbook (ask [Danny Chiao](https://tectonfeast.slack.com/team/U029405HFEU) in Slack for access): +1. Create a new space within the Feast collection +2. Go to the overflow menu on the top -> Synchronize with Git + 1. Specify GitHub as the provider + + ![](new_branch_part_1.png) + 2. Configure to point to the new release branch + + ![](new_branch_part_2.png) +3. Publish the new page for this branch as part of the collection + + ![](new_branch_part_3.png) +4. Go back to the main Feast collection and go to the overflow menu -> "Customize collection" + + ![](new_branch_part_3.png) +5. Configure the default space to be your new branch and save + + ![](new_branch_part_5.png) +6. Verify on docs.feast.dev that this new space is the default (this may take a few minutes to propagate, and your browser cache may be caching the old branch as the default) \ No newline at end of file diff --git a/docs/project/versioning-policy.md b/docs/project/versioning-policy.md index 8e51676355..b1ff2c75e7 100644 --- a/docs/project/versioning-policy.md +++ b/docs/project/versioning-policy.md @@ -23,24 +23,18 @@ In general, unless you're committing code that only applies to a particular rele The following table shows the **status** \(stable, beta, or alpha\) of Feast components. -Application status indicators for Feast: +Component status indicators for Feast: * **Stable** means that the component has reached a sufficient level of stability and adoption that the Feast community has deemed the component stable. Please see the stability criteria below. * **Beta** means that the component is working towards a version 1.0 release. Beta does not mean a component is unstable, it simply means the component has not met the full criteria of stability. * **Alpha** means that the component is in the early phases of development and/or integration into Feast. -| Application | Status | Notes | -| :--- | :--- | :--- | -| [Feast Serving](https://github.com/feast-dev/feast-java) | Beta | APIs are considered stable and will not have breaking changes within 3 minor versions. | -| [Feast Core](https://github.com/feast-dev/feast-java) | Beta | At risk of deprecation | -| [Feast Java Client](https://github.com/feast-dev/feast-java) | Beta | | -| [Feast Python SDK](https://github.com/feast-dev/feast) | Beta | | -| [Feast Go Client](https://github.com/feast-dev/feast) | Beta | | -| [Feast Spark Python SDK](https://github.com/feast-dev/feast-spark) | Alpha | | -| [Feast Spark Launchers](https://github.com/feast-dev/feast-spark) | Alpha | | -| [Feast Job Service](https://github.com/feast-dev/feast-spark) | Alpha | Scheduled for deprecation | -| [Feast Helm Chart](https://github.com/feast-dev/feast-helm-charts) | Beta | | -| | | | +| Component | Status | Notes | +|:---------------------------------------------------------------------------------|:-------| :--- | +| [Feast Python SDK](https://github.com/feast-dev/feast/tree/master/sdk/python) | Stable | | +| [Feast Go Feature Server](https://github.com/feast-dev/feast/tree/master/) | Beta | | +| [Feast Java Feature Server](https://github.com/feast-dev/feast/tree/master/java) | Alpha | | +| | | | Criteria for reaching _**stable**_ status: diff --git a/docs/roadmap.md b/docs/roadmap.md index 8461256a15..ae46eb9005 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -4,7 +4,7 @@ The list below contains the functionality that contributors are planning to deve * Items below that are in development (or planned for development) will be indicated in parentheses. * We welcome contribution to all items in the roadmap! -* Want to speak to a Feast contributor? We are more than happy to jump on a call. Please schedule a time using [Calendly](https://calendly.com/d/x2ry-g5bb/meet-with-feast-team). +* Have questions about the roadmap? Go to the Slack channel to ask on #feast-development * **Data Sources** * [x] [Snowflake source](https://docs.feast.dev/reference/data-sources/snowflake) @@ -16,7 +16,6 @@ The list below contains the functionality that contributors are planning to deve * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres) * [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark) * [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push)) - * [ ] HTTP source * **Offline Stores** * [x] [Snowflake](https://docs.feast.dev/reference/offline-stores/snowflake) * [x] [Redshift](https://docs.feast.dev/reference/offline-stores/redshift) @@ -49,26 +48,17 @@ The list below contains the functionality that contributors are planning to deve * **Deployments** * [x] AWS Lambda (Alpha release. See [RFC](https://docs.google.com/document/d/1eZWKWzfBif66LDN32IajpaG-j82LSHCCOzY6R7Ax7MI/edit)) * [x] Kubernetes (See [guide](https://docs.feast.dev/how-to-guides/running-feast-in-production#4.3.-java-based-feature-server-deployed-on-kubernetes)) - * [ ] Cloud Run - * [ ] KNative * **Feature Serving** * [x] Python Client - * [x] REST Feature Server (Python) (Alpha release. See [RFC](https://docs.google.com/document/d/1iXvFhAsJ5jgAhPOpTdB3j-Wj1S9x3Ev\_Wr6ZpnLzER4/edit)) - * [x] gRPC Feature Server (Java) (See [#1497](https://github.com/feast-dev/feast/issues/1497)) - * [x] Push API - * [ ] Java Client - * [ ] Go Client - * [ ] Delete API - * [ ] Feature Logging (for training) + * [x] REST Feature Server (Python) (See [RFC](https://docs.google.com/document/d/1iXvFhAsJ5jgAhPOpTdB3j-Wj1S9x3Ev\_Wr6ZpnLzER4/edit)) + * [x] REST / gRPC Feature Server (Go) (Alpha release. See [docs](https://docs.feast.dev/reference/feature-servers/go-feature-retrieval) + * [x] gRPC Feature Server (Java) (Alpha release. See [#1497](https://github.com/feast-dev/feast/issues/1497)) * **Data Quality Management (See [RFC](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit))** * [x] Data profiling and validation (Great Expectations) - * [ ] Training-serving skew detection (in progress) - * [ ] Metric production - * [ ] Drift detection * **Feature Discovery and Governance** * [x] Python SDK for browsing feature registry * [x] CLI for browsing feature registry * [x] Model-centric feature tracking (feature services) * [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py)) - * [x] Feast Web UI (Alpha release. See [documentation](https://docs.feast.dev/reference/alpha-web-ui)) - * [ ] REST API for browsing feature registry + * [x] DataHub integration (see [DataHub Feast docs](https://datahubproject.io/docs/generated/ingestion/sources/feast/)) + * [x] Feast Web UI (Alpha release. See [documentation](https://docs.feast.dev/reference/alpha-web-ui)) \ No newline at end of file