diff --git a/docs/README.md b/docs/README.md index aa0be6090d..9554bbcb2e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,12 +1,12 @@ # Introduction -Feast \(**Fea**ture **St**ore\) is a tool for managing and serving machine learning features. +### What is Feast? -> Feast is the bridge between your models and your data +Feast \(**Fea**ture **St**ore\) is a tool for managing and serving machine learning features. Feast aims to: -* Provide a unified means of managing feature data from a single person to large enterprises. +* Provide a unified means of managing feature data, whether you are a single user or a large enterprise. * Provide scalable and performant access to feature data when training and serving models. * Provide consistent and point-in-time correct access to feature data. * Enable discovery, documentation, and insights into your features. @@ -17,3 +17,19 @@ Feast decouples feature engineering from feature usage. Features that are added This means that new ML projects start with a process of feature selection from a catalog instead of having to do feature engineering from scratch. +### How can I get started? + +{% hint style="info" %} +The best way to learn Feast is to use it. Jump over to our [Quickstart](quickstart.md) guide to have one of our examples running in no time at all! +{% endhint %} + + Explore the following resources to get started with Feast: + +* The [Getting Started](getting-started/) section provides guides on [Installing Feast](getting-started/install-feast/) and [Connecting to Feast](getting-started/connect-to-feast/). +* The [Concepts](./) section describes all important Feast API concepts. +* The [User guide](user-guide/data-ingestion.md) section provides guidance on completing Feast workflows. +* The [Examples](https://github.com/feast-dev/feast/tree/master/examples) section contains Jupyter notebooks that you can run on your Feast deployment. +* The [Advanced](advanced/stores.md) section contains information about both advanced and operational aspects of Feast. +* The [Reference](reference/api/) section contains detailed API and design documents for advanced users. +* The [Contributing](contributing/contributing.md) section contains resources for anyone who wants to contribute to Feast. + diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 21de0c8a94..8b8434be5c 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -1,15 +1,15 @@ # Table of contents * [Introduction](README.md) -* [Why Feast?](why-feast.md) * [Quickstart](quickstart.md) * [Getting Started](getting-started/README.md) - * [Deploying Feast](getting-started/deploying-feast/README.md) - * [Kubernetes](getting-started/deploying-feast/kubernetes.md) - * [Kubernetes \(Amazon EKS\) with Terraform](getting-started/deploying-feast/kubernetes-amazon-eks-with-terraform.md) - * [Connecting to Feast](getting-started/connecting-to-feast-1/README.md) - * [Python SDK](getting-started/connecting-to-feast-1/python-sdk.md) - * [Feast CLI](getting-started/connecting-to-feast-1/connecting-to-feast.md) + * [Install Feast](getting-started/install-feast/README.md) + * [Kubernetes \(with Helm\)](getting-started/install-feast/kubernetes-with-helm.md) + * [Amazon EKS \(with Terraform\)](getting-started/install-feast/kubernetes-amazon-eks-with-terraform.md) + * [Connect to Feast](getting-started/connect-to-feast/README.md) + * [Python SDK](getting-started/connect-to-feast/python-sdk.md) + * [Feast CLI](getting-started/connect-to-feast/connecting-to-feast.md) + * [Learn Feast](getting-started/learn-feast.md) * [Roadmap](roadmap.md) * [Changelog](https://github.com/feast-dev/feast/blob/master/CHANGELOG.md) * [Community](getting-help.md) @@ -44,15 +44,15 @@ ## Reference -* [Metrics Reference](reference/metrics-reference.md) -* [Configuration Reference](reference/configuration-reference.md) -* [API, Supported Versions & Deprecation](reference/api-supported-versions-and-deprecation.md) * [API Reference](reference/api/README.md) * [Core gRPC API](https://api.docs.feast.dev/grpc/feast.core.pb.html) * [Serving gRPC API](https://api.docs.feast.dev/grpc/feast.serving.pb.html) * [gRPC Types](https://api.docs.feast.dev/grpc/feast.types.pb.html) +* [Configuration Reference](reference/configuration-reference/README.md) * [Go SDK](https://godoc.org/github.com/feast-dev/feast/sdk/go) +* [Metrics Reference](reference/metrics-reference/README.md) * [Java SDK](https://javadoc.io/doc/dev.feast/feast-sdk) +* [API, Supported Versions & Deprecation](reference/api-supported-versions-and-deprecation/README.md) * [Python SDK](https://api.docs.feast.dev/python/) * [Limitations](reference/limitations.md) * [Design Decisions](reference/design-decisions.md) diff --git a/docs/advanced/metrics.md b/docs/advanced/metrics.md index f0e7b8c89d..2c91a98b0e 100644 --- a/docs/advanced/metrics.md +++ b/docs/advanced/metrics.md @@ -7,7 +7,7 @@ Feast Components export metrics that can provide insight into Feast behavior: * [Feast Ingestion Jobs can be configured to push metrics into StatsD](metrics.md#2-exporting-feast-metrics-to-prometheus) * [Prometheus can be configured to scrap metrics from Feast Core and Serving.](metrics.md#2-exporting-feast-metrics-to-prometheus) -See the [Metrics Reference ](../reference/metrics-reference.md)for documentation on metrics are exported by Feast. +See the [Metrics Reference ](../reference/metrics-reference/)for documentation on metrics are exported by Feast. {% hint style="info" %} Feast Job Controller currently does not export any metrics on its own. However its `application.yml` is used to configure metrics export for ingestion jobs. @@ -51,7 +51,7 @@ server: ## 3. Futher Reading. -See the [Metrics Reference ](../reference/metrics-reference.md)for documentation on metrics are exported by Feast. +See the [Metrics Reference ](../reference/metrics-reference/)for documentation on metrics are exported by Feast. ## diff --git a/docs/advanced/security.md b/docs/advanced/security.md index a8f0ca8fa7..937dadb3bd 100644 --- a/docs/advanced/security.md +++ b/docs/advanced/security.md @@ -40,7 +40,7 @@ The following properties configure SSL/TLS. These properties are located in thei ### Configuring SSL/TLS on Python SDK/CLI -To enable SSL/TLS in the [Feast Python SDK](https://api.docs.feast.dev/python/#feast.client.Client) or [Feast CLI](../getting-started/connecting-to-feast-1/connecting-to-feast.md), set the config options via `feast config`: +To enable SSL/TLS in the [Feast Python SDK](https://api.docs.feast.dev/python/#feast.client.Client) or [Feast CLI](../getting-started/connect-to-feast/connecting-to-feast.md), set the config options via `feast config`: | Configuration Option | Description | | :--- | :--- | @@ -188,7 +188,7 @@ OAuth Provider makes an OAuth [client credentials](https://auth0.com/docs/flows/ ### **Enabling Authentication in Python SDK/CLI** -Configure the [Feast Python SDK](https://api.docs.feast.dev/python/) and [Feast CLI](../getting-started/connecting-to-feast-1/connecting-to-feast.md) to use authentication via `feast config`: +Configure the [Feast Python SDK](https://api.docs.feast.dev/python/) and [Feast CLI](../getting-started/connect-to-feast/connecting-to-feast.md) to use authentication via `feast config`: ```python $ feast config set enable_auth true diff --git a/docs/advanced/upgrading.md b/docs/advanced/upgrading.md index 4fe83bc092..8e1c426584 100644 --- a/docs/advanced/upgrading.md +++ b/docs/advanced/upgrading.md @@ -22,7 +22,7 @@ Feast now prevents feature sets from being applied if no store is subscribed to ### Feast Core's Job Coordinator is now Feast Job Controller -In v0.7, Feast Core's Job Coordinator has been decoupled from Feast Core and runs as a separate Feast Job Controller application. See its [Configuration reference](../reference/configuration-reference.md#2-feast-core-serving-and-job-controller) for how to configure Feast Job Controller. +In v0.7, Feast Core's Job Coordinator has been decoupled from Feast Core and runs as a separate Feast Job Controller application. See its [Configuration reference](../reference/configuration-reference/#2-feast-core-serving-and-job-controller) for how to configure Feast Job Controller. **Ingestion Job API** diff --git a/docs/concepts/architecture.md b/docs/concepts/architecture.md index 9176f03d47..8b462bcbfa 100644 --- a/docs/concepts/architecture.md +++ b/docs/concepts/architecture.md @@ -14,7 +14,7 @@ Feast Core is the central management service of a Feast deployment. It's role is Before you ingest data into Feast, first register one or more entity, then register feature tables. These [feature tables](feature-tables.md) tell Feast where to find their data and how to ingest it. The feature tables also describe the characteristics of the data for validation purposes. After a feature table is registered, you can start a Spark job to populate a store with data from the defined source in the feature table specification. -To ensure stores are populated with data, you must publish the data to a [source](sources.md). Currently, Feast supports a few batch and stream sources. Feast users \(or pipelines\) ingest batch data through the [Feast Python SDK](../getting-started/connecting-to-feast-1/python-sdk.md) using its `ingest()` method. The SDK publishes the data into the batch source specified for the feature table's batch source. +To ensure stores are populated with data, you must publish the data to a [source](sources.md). Currently, Feast supports a few batch and stream sources. Feast users \(or pipelines\) ingest batch data through the [Feast Python SDK](../getting-started/connect-to-feast/python-sdk.md) using its `ingest()` method. The SDK publishes the data into the batch source specified for the feature table's batch source. Streaming systems can also ingest data into Feast. This is done by publishing to the correct stream source from the feature table specification in the expected format. The topic and brokers can be found on the feature table's stream source if specified during registration. diff --git a/docs/contributing/release-process.md b/docs/contributing/release-process.md index a331312805..533d68b916 100644 --- a/docs/contributing/release-process.md +++ b/docs/contributing/release-process.md @@ -38,7 +38,7 @@ For Feast maintainers, these are the concrete steps for making a new release. 3. Check that versions are updated with `env TARGET_MERGE_BRANCH=master make lint-versions` 7. Create a [GitHub release](https://github.com/feast-dev/feast/releases) which includes a summary of im~~p~~ortant changes as well as any artifacts associated with the release. Make sure to include the same change log as added in [CHANGELOG.md](https://github.com/feast-dev/feast/blob/master/CHANGELOG.md). Use `Feast vX.Y.Z` as the title. 8. Update the[ Upgrade Guide](../advanced/upgrading.md) to include the action required instructions for users to upgrade to this new release. Instructions should include a migration for each breaking change made to this release. -9. Update[ Feast Supported Versions](../reference/api-supported-versions-and-deprecation.md#3-supported-versions) to include the supported versions of each component. +9. Update[ Feast Supported Versions](../reference/api-supported-versions-and-deprecation/#3-supported-versions) to include the supported versions of each component. When a tag that matches a Semantic Version string is pushed, CI will automatically build and push the relevant artifacts to their repositories or package managers \(docker images, Python wheels, etc\). JVM artifacts are promoted from Sonatype OSSRH to Maven Central, but it sometimes takes some time for them to be available. The `sdk/go/v tag` is required to version the Go SDK go module so that users can go get a specific tagged release of the Go SDK. @@ -75,8 +75,8 @@ docker run -it --rm ferrarimarco/github-changelog-generator \ ### Flag Breaking Changes & Deprecations -It's important to flag breaking changes and deprecation to the API for each release so that we can maintain [API compatibility](../reference/api-supported-versions-and-deprecation.md#2-api-compatibility). +It's important to flag breaking changes and deprecation to the API for each release so that we can maintain [API compatibility](../reference/api-supported-versions-and-deprecation/#2-api-compatibility). * Developers should have flagged PRs with breaking changes with the `compat/breaking` label. However, it's important to double check each PR's release notes and contents for changes that will break [API compatibility](https://app.gitbook.com/@feast/s/docs/~/drafts/-MGstXoieRglzkPw_d1Q/v/master/reference/api-supported-versions-and-deprecation#2-api-compatibility) and manually label `compat/breaking` to PRs with undeclared breaking changes. The change log will have to be regenerated if any new labels have to be added. -* Record any deprecation in [Deprecations Page](../reference/api-supported-versions-and-deprecation.md#4-deprecations). +* Record any deprecation in [Deprecations Page](../reference/api-supported-versions-and-deprecation/#4-deprecations). diff --git a/docs/getting-help.md b/docs/getting-help.md index 32b21461b1..e9ead181ca 100644 --- a/docs/getting-help.md +++ b/docs/getting-help.md @@ -1,36 +1,29 @@ # Community -## Chat +### Links & Resources -* Come and say hello in [\#Feast](https://join.slack.com/t/kubeflow/shared_invite/zt-cpr020z4-PfcAue_2nw67~iIDy7maAQ) over in the Kubeflow Slack. +* [Slack](https://kubeflow.slack.com/messages/CE0L8T267): We use the channel [\#Feast](https://kubeflow.slack.com/messages/CE0L8T267) in [kubeflow.slack.com](https://join.slack.com/t/kubeflow/shared_invite/zt-cpr020z4-PfcAue_2nw67~iIDy7maAQ). Feel free to ask questions or say hello! +* [Mailing list](https://groups.google.com/d/forum/feast-dev): We have both a user and developer mailing list. + * Feast users should join [feast-discuss@googlegroups.com](mailto:feast-discuss@googlegroups.com) group by clicking [here](https://groups.google.com/g/feast-discuss). + * Feast developers should join [feast-dev@googlegroups.com](mailto:feast-dev@googlegroups.com) group by clicking [here](https://groups.google.com/d/forum/feast-dev). +* [Google Drive](https://drive.google.com/drive/u/0/folders/0AAe8j7ZK3sxSUk9PVA): The above groups will also grant access to our public [Feast Google Drive](https://drive.google.com/drive/u/0/folders/0AAe8j7ZK3sxSUk9PVA). The drive is used as a central repository for all Feast resources. For example: + * Design proposals in the form of Request for Comments \(RFC\). + * User surveys and meeting minutes. + * Slide decks of conferences our contributors have spoken at. +* [Feast GitHub Repository](https://github.com/feast-dev/feast/): Find the complete Feast codebase on GitHub. -## GitHub +### How can I get help? -* Feast's GitHub repo can be [found here](https://github.com/feast-dev/feast/). -* Found a bug or need a feature? [Create an issue on GitHub](https://github.com/feast-dev/feast/issues/new) +* **Slack:** Need to speak to a human? Come ask a question in our Slack channel \(link above\) +* **GitHub Issues:** Found a bug or need a feature? [Create an issue on GitHub](https://github.com/feast-dev/feast/issues/new). +* **StackOverflow:** Need to ask a question on how to use Feast? We also monitor and respond to [StackOverflow](https://stackoverflow.com/questions/tagged/feast). -## Community Call +### Community Call -We have a community call every 2 weeks. Alternating between two times. +We have a community call every 2 weeks. Alternating between two times: * 11 am \(UTC + 8\) * 5 pm \(UTC + 8\) -Please join the [feast-dev](getting-help.md#feast-development) mailing list to receive the the calendar invitation. - -## Mailing list - -### Feast discussion - -* Google Group: [https://groups.google.com/d/forum/feast-discuss](https://groups.google.com/d/forum/feast-discuss) -* Mailing List: [feast-discuss@googlegroups.com](mailto:feast-discuss@googlegroups.com) - -### Feast development - -* Google Group: [https://groups.google.com/d/forum/feast-dev](https://groups.google.com/d/forum/feast-dev) -* Mailing List: [feast-dev@googlegroups.com](mailto:feast-dev@googlegroups.com) - -## Google Drive - -The Feast community also maintains a [Google Drive](https://drive.google.com/drive/u/0/folders/0AAe8j7ZK3sxSUk9PVA) with documents like RFCs, meeting notes, or roadmaps. Please join one of the above mailing lists \(feast-dev or feast-discuss\) to gain access to the drive. +Join the [feast-dev](getting-help.md#feast-development) mailing list to receive a Google calendar invitation. diff --git a/docs/getting-started/README.md b/docs/getting-started/README.md index c1837bf00a..4a5296eba3 100644 --- a/docs/getting-started/README.md +++ b/docs/getting-started/README.md @@ -1,14 +1,20 @@ # Getting Started -If you would like to learn more about Feast concepts and its architecture, see the `User Guide` +### Install Feast -If you would like to connect to an existing Feast deployment, click on `Connecting to Feast` +If you would like to deploy a new installation of Feast, click on [Install Feast](install-feast/) -{% page-ref page="connecting-to-feast-1/" %} +{% page-ref page="install-feast/" %} -If you would like to deploy a new installation of Feast, click on `Deploying Feast` +### Connect to Feast -{% page-ref page="deploying-feast/" %} +If you would like to connect to an existing Feast deployment, click on [Connect to Feast](connect-to-feast/) -If you are connected to a running Feast deployment, have a look at our [example tutorials](https://github.com/gojek/feast/tree/master/examples). +{% page-ref page="connect-to-feast/" %} + +### Learn Feast + +If you would like to learn more about Feast, click on [Learn Feast](learn-feast.md) + +{% page-ref page="learn-feast.md" %} diff --git a/docs/getting-started/connect-to-feast/README.md b/docs/getting-started/connect-to-feast/README.md new file mode 100644 index 0000000000..214cf89fa3 --- /dev/null +++ b/docs/getting-started/connect-to-feast/README.md @@ -0,0 +1,31 @@ +# Connect to Feast + +### Feast Python SDK + +The Feast Python SDK is used as a library to interact with a Feast deployment. + +* Define, register, and manage entities and features +* Ingest data into Feast +* Build and retrieve training datasets +* Retrieve online features + +{% page-ref page="python-sdk.md" %} + +### Feast CLI + +The Feast CLI is a command line implementation of the Feast Python SDK. + +* Define, register, and manage entities and features from the terminal +* Ingest data into Feast +* Manage ingestion jobs + +{% page-ref page="connecting-to-feast.md" %} + +### Online Serving Clients + +The following clients can be used to retrieve online feature values: + +* [Feast Python SDK](https://api.docs.feast.dev/python/) +* [Feast Go SDK](https://godoc.org/github.com/feast-dev/feast/sdk/go) +* [Feast Java SDK](https://javadoc.io/doc/dev.feast/feast-sdk) + diff --git a/docs/getting-started/connecting-to-feast-1/connecting-to-feast.md b/docs/getting-started/connect-to-feast/connecting-to-feast.md similarity index 100% rename from docs/getting-started/connecting-to-feast-1/connecting-to-feast.md rename to docs/getting-started/connect-to-feast/connecting-to-feast.md diff --git a/docs/getting-started/connecting-to-feast-1/python-sdk.md b/docs/getting-started/connect-to-feast/python-sdk.md similarity index 86% rename from docs/getting-started/connecting-to-feast-1/python-sdk.md rename to docs/getting-started/connect-to-feast/python-sdk.md index 2dd9abec2d..bf31bd3849 100644 --- a/docs/getting-started/connecting-to-feast-1/python-sdk.md +++ b/docs/getting-started/connect-to-feast/python-sdk.md @@ -6,7 +6,7 @@ Install the [Feast Python SDK](https://api.docs.feast.dev/python/) using pip: pip install feast ``` -You can then connect to an existing Feast deployment: +Connect to an existing Feast Core deployment: ```python from feast import Client @@ -18,5 +18,3 @@ client = Client(core_url='feast.example.com:6565') client.list_feature_tables() ``` - - diff --git a/docs/getting-started/connecting-to-feast-1/README.md b/docs/getting-started/connecting-to-feast-1/README.md deleted file mode 100644 index 1bea45d7be..0000000000 --- a/docs/getting-started/connecting-to-feast-1/README.md +++ /dev/null @@ -1,27 +0,0 @@ -# Connecting to Feast - -### Feast Python SDK - -* Define, register, and manage entities and features -* Ingest data into Feast -* Build and retrieve training datasets -* Retrieve online features - -{% page-ref page="python-sdk.md" %} - -### Feast CLI - -* Define, register, and manage entities and features from the terminal -* Ingest data into Feast -* Manage ingestion jobs - -{% page-ref page="connecting-to-feast.md" %} - -### Online Serving Clients - -Feast provides three ways to retrieve data in production for online serving: - -* [Python SDK](https://api.docs.feast.dev/python/) -* [Go Client library](https://godoc.org/github.com/feast-dev/feast/sdk/go) -* [Java Client library](https://javadoc.io/doc/dev.feast/feast-sdk) - diff --git a/docs/getting-started/deploying-feast/README.md b/docs/getting-started/deploying-feast/README.md deleted file mode 100644 index ff58fddce0..0000000000 --- a/docs/getting-started/deploying-feast/README.md +++ /dev/null @@ -1,8 +0,0 @@ -# Deploying Feast - -{% page-ref page="kubernetes.md" %} - -{% page-ref page="kubernetes-amazon-eks-with-terraform.md" %} - - - diff --git a/docs/getting-started/install-feast/README.md b/docs/getting-started/install-feast/README.md new file mode 100644 index 0000000000..d86f4d8c8d --- /dev/null +++ b/docs/getting-started/install-feast/README.md @@ -0,0 +1,14 @@ +# Install Feast + +### Kubernetes \(with Helm\) + +This guide installs Feast into an existing Kubernetes cluster using Helm. The installation is **not** specific to any cloud platform or environment, but **does** require Kubernetes and Helm. + +{% page-ref page="kubernetes-with-helm.md" %} + +### Amazon EKS \(with Terraform\) + +This guide installs Feast into an AWS environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly. + +{% page-ref page="kubernetes-amazon-eks-with-terraform.md" %} + diff --git a/docs/getting-started/deploying-feast/kubernetes-amazon-eks-with-terraform.md b/docs/getting-started/install-feast/kubernetes-amazon-eks-with-terraform.md similarity index 94% rename from docs/getting-started/deploying-feast/kubernetes-amazon-eks-with-terraform.md rename to docs/getting-started/install-feast/kubernetes-amazon-eks-with-terraform.md index a2b5cd1c14..d3058913d1 100644 --- a/docs/getting-started/deploying-feast/kubernetes-amazon-eks-with-terraform.md +++ b/docs/getting-started/install-feast/kubernetes-amazon-eks-with-terraform.md @@ -1,4 +1,4 @@ -# Kubernetes \(Amazon EKS\) with Terraform +# Amazon EKS \(with Terraform\) ### Overview @@ -38,7 +38,7 @@ region = "us-east-1" ### 3. Apply -After completing the configuration, initiate Terraform and apply: +After completing the configuration, initialize Terraform and apply: ```bash $ cd feast/infra/terraform/aws @@ -64,5 +64,5 @@ Forwarding from 127.0.0.1:8888 -> 8888 Forwarding from [::1]:8888 -> 8888 ``` -You can now connect to the bundled Jupyter Notebook Server at `localhost:8888` and follow the example notebooks. +You can now connect to the bundled Jupyter Notebook Server at `localhost:8888` and follow the example Jupyter notebook. diff --git a/docs/getting-started/deploying-feast/kubernetes.md b/docs/getting-started/install-feast/kubernetes-with-helm.md similarity index 66% rename from docs/getting-started/deploying-feast/kubernetes.md rename to docs/getting-started/install-feast/kubernetes-with-helm.md index 4427aa88f9..1f38a8ba50 100644 --- a/docs/getting-started/deploying-feast/kubernetes.md +++ b/docs/getting-started/install-feast/kubernetes-with-helm.md @@ -1,4 +1,4 @@ -# Kubernetes +# Kubernetes \(with Helm\) ### Overview @@ -8,28 +8,26 @@ This guide installs Feast on an existing Kubernetes cluster, and ensures the fol * Feast Online Serving * Postgres * Redis -* Feast Jupyter -* Prometheus - -This guide also contains instructions for customizing your installation for different production use cases. +* Feast Jupyter \(Optional\) +* Prometheus \(Optional\) ### 1. Requirements 1. Install and configure [Kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) -2. Install [Helm ](https://helm.sh/)3 +2. Install [Helm 3](https://helm.sh/) ### 2. Preparation -Add Feast Helm repository and download the latest charts: +Add the Feast Helm repository and download the latest charts: ```text helm repo add feast-charts https://feast-charts.storage.googleapis.com helm repo update ``` -Feast includes a Helm chart that installs all necessary components to run `Feast Core`, `Feast Online Serving`, and the example Jupyter notebook. +Feast includes a Helm chart that installs all necessary components to run Feast Core, Feast Online Serving, and an example Jupyter notebook. -Because `Feast Core` requires Postgres to run, create a Kubernetes secret for the credential: +Feast Core requires Postgres to run, which requires a secret to be set on Kubernetes: ```bash kubectl create secret generic feast-postgresql --from-literal=postgresql-password=password @@ -37,7 +35,7 @@ kubectl create secret generic feast-postgresql --from-literal=postgresql-passwor ### 3. Installation -Install Feast using Helm. The pods take a minute to initiate, while at the same time Feast Online Serving restarts until Feast Core has launched. +Install Feast using Helm. The pods may take a few minutes to initialize. ```bash helm install feast-release feast-charts/feast @@ -45,7 +43,7 @@ helm install feast-release feast-charts/feast ### 4. Use Jupyter to connect to Feast -After all the pods are ready, connect to the Jupyter Notebook Server running in the cluster: +After all the pods are in a `RUNNING` state, port-forward to the Jupyter Notebook Server in the cluster: ```bash kubectl port-forward \ @@ -57,7 +55,7 @@ Forwarding from 127.0.0.1:8888 -> 8888 Forwarding from [::1]:8888 -> 8888 ``` -You can now connect to the bundled Jupyter Notebook Server at `localhost:8888` and follow the example notebooks. +You can now connect to the bundled Jupyter Notebook Server at `localhost:8888` and follow the example Jupyter notebook. {% embed url="http://localhost:8888/tree?" caption="" %} @@ -66,7 +64,5 @@ You can now connect to the bundled Jupyter Notebook Server at `localhost:8888` a * [Feast Concepts](../../concepts/overview.md) * [Feast Examples/Tutorials](https://github.com/feast-dev/feast/tree/master/examples) * [Feast Helm Chart Documentation](https://github.com/feast-dev/feast/blob/master/infra/charts/feast/README.md) -* [Configuring Feast components](../../reference/configuration-reference.md) - - +* [Configuring Feast components](../../reference/configuration-reference/) diff --git a/docs/getting-started/learn-feast.md b/docs/getting-started/learn-feast.md new file mode 100644 index 0000000000..d067b47e60 --- /dev/null +++ b/docs/getting-started/learn-feast.md @@ -0,0 +1,15 @@ +# Learn Feast + +Explore the following resources to learn more about Feast: + +* The [Concepts](../) section describes all important Feast API concepts. +* The [User guide](../user-guide/data-ingestion.md) section provides guidance on completing Feast workflows. +* The [Examples](https://github.com/feast-dev/feast/tree/master/examples) section contains Jupyter notebooks that you can run on your Feast deployment. +* The [Advanced](../advanced/stores.md) section contains information about both advanced and operational aspects of Feast. +* The [Reference](../reference/api/) section contains detailed API and design documents for advanced users. +* The [Contributing](../contributing/contributing.md) section contains resources for anyone who wants to contribute to Feast. + +{% hint style="info" %} +The best way to learn Feast is to use it. Jump over to our [Quickstart](../quickstart.md) guide to have one of our examples running in no time at all! +{% endhint %} + diff --git a/docs/quickstart.md b/docs/quickstart.md index 50c5c2295c..0e6bd9f788 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -2,59 +2,79 @@ ### Overview -This brief introduction shows you how to deploy Feast using [Docker Compose](https://docs.docker.com/get-started/). Docker Compose enables you to explore the functionality provided by Feast while requiring only minimal infrastructure. It includes a built in Jupyter Notebook Server that is preloaded with PySpark and the Feast Python SDK, as well as Feast example notebooks to get you started. +This guide shows you how to deploy Feast using [Docker Compose](https://docs.docker.com/get-started/). Docker Compose enables you to explore the functionality provided by Feast while requiring only minimal infrastructure. + +This guide includes the following containerized components: + +* A complete Feast deployment \(Feast Core with Postgres, Feast Online Serving with Redis\). +* A Jupyter Notebook Server with built in Feast example\(s\). +* A Kafka cluster for testing streaming ingestion. ### 1. Requirements -* Install [Docker Compose](https://docs.docker.com/compose/install/) -* Optional dependencies: - * a [GCP service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) that has access to [Google Cloud Storage](https://cloud.google.com/storage) +1. Install [Docker Compose](https://docs.docker.com/compose/install/) ### 2. Configure Environment -Before starting, clone the latest stable version of Feast from the [Feast repository](https://github.com/gojek/feast/), and configure. +Clone the latest stable version of Feast from the [Feast repository](https://github.com/gojek/feast/): ```text git clone https://github.com/feast-dev/feast.git cd feast/infra/docker-compose +``` + +Create a new configuration file: + +```text cp .env.sample .env ``` ### 3. Start Feast Services -Start the Feast services. Ensure that the following ports are free on the host machines: `6565`, `6566`, `8888`, `9094`, `5432`. Alternatively, change port mapping to use a different port on the host. +Start Feast with Docker Compose: -```javascript +```text docker-compose up -d ``` -{% hint style="info" %} -Docker Compose takes a minute to launch: +Wait until all all containers are in a running state: -* While the deployment is initiating, Feast Online Serving container may restart. Once Feast Core has launched, any container restarts should be auto-corrected. -* If container restarts continue for more than 10 minutes, check the Docker Compose log for any errors that prevent Feast Core from starting successfully. -{% endhint %} +```text +docker-compose ps +``` -You can now connect to the bundled Jupyter Notebook Server at `localhost:8888` and follow the example notebooks. +You can now connect to the bundled Jupyter Notebook Server running at `localhost:8888` and follow the example Jupyter notebook. {% embed url="http://localhost:8888/tree?" caption="" %} -### 4. Optional Dependencies +### 4. Troubleshooting + +#### 4.1 Open ports + +Please ensure that the following ports are available on your host machine: `6565`, `6566`, `8888`, `9094`, `5432`. + +If a port conflict cannot be resolved, you can modify all port mappings in the provided [docker-compose.yml](https://github.com/feast-dev/feast/tree/master/infra/docker-compose) file to use a different port on the host. + +#### 4.2 Containers are restarting or unavailable + +If some of the containers continue to restart, or you are unable to access a service, inspect the logs using the following command: + +```javascript +docker-compose logs -f -t +``` -#### 4.1 Configure Google Cloud Platform \(GCP\) +If you are unable to resolve the problem, visit [GitHub](https://github.com/feast-dev/feast/issues) to create an issue. -By default, the example Jupyter notebook does not require any GCP dependencies. If you modify the example to require GCP services \(for example Google Cloud Storage\), you need to establish a [service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) that is associated with the Jupyter notebook. +### 5. Configuration -{% hint style="warning" %} -Confirm that the GCP service account has sufficient privileges to access the required GCP services. -{% endhint %} +The Feast Docker Compose setup can be configured by modifying properties in your `.env` file. -After you create the service account, download the associated JSON key file and copy the file to the path configured in `.env` under `GCP_SERVICE_ACCOUNT` . +#### 5.1 Accessing Google Cloud Storage \(GCP\) -### 5. Further Reading +To access Google Cloud Storage as a data source, the Docker Compose installation requires access to a GCP service account. -* [Feast Concepts](concepts/overview.md) -* [Feast Examples/Tutorials](https://github.com/feast-dev/feast/tree/master/examples) -* [Configuring Feast Components](reference/configuration-reference.md) -* [Configuration Reference](https://app.gitbook.com/@feast/s/docs/v/master/reference/configuration-reference) +* Create a new [service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) and save a JSON key. +* Grant the service account access to your bucket\(s\). +* Copy the service account to the path you have configured in `.env` under `GCP_SERVICE_ACCOUNT`. +* Restart your Docker Compose setup of Feast. diff --git a/docs/reference/api-supported-versions-and-deprecation.md b/docs/reference/api-supported-versions-and-deprecation/README.md similarity index 93% rename from docs/reference/api-supported-versions-and-deprecation.md rename to docs/reference/api-supported-versions-and-deprecation/README.md index 8a9f8bba9b..4636b40783 100644 --- a/docs/reference/api-supported-versions-and-deprecation.md +++ b/docs/reference/api-supported-versions-and-deprecation/README.md @@ -8,10 +8,10 @@ description: 'Tracking Feast''s API Compatibility, support Feast versions, Depre This document tracks Feast's API compatibility. Users are dependent on Feast providing a stable, compatible API such that Feast upgrades do not significantly impact the systems that they have built on Feast, while developers have to make changes to correct API design decisions that no longer make sense. By tracking Feast's API Compatibility, support Feast versions, this document attempts to find common ground between the two needs. -1. [Release Versioning](api-supported-versions-and-deprecation.md#1-release-versioning) -2. [API Compatibility Policy](api-supported-versions-and-deprecation.md#2-api-compatibility) -3. [Supported Feast Versions.](api-supported-versions-and-deprecation.md#3-supported-versions) -4. [Deprecations.](api-supported-versions-and-deprecation.md#4-deprecations) +1. [Release Versioning](./#1-release-versioning) +2. [API Compatibility Policy](./#2-api-compatibility) +3. [Supported Feast Versions.](./#3-supported-versions) +4. [Deprecations.](./#4-deprecations) ## 1. Release Versioning diff --git a/docs/reference/configuration-reference.md b/docs/reference/configuration-reference/README.md similarity index 84% rename from docs/reference/configuration-reference.md rename to docs/reference/configuration-reference/README.md index 7fc2bf2e61..f67f0c0937 100644 --- a/docs/reference/configuration-reference.md +++ b/docs/reference/configuration-reference/README.md @@ -4,9 +4,9 @@ This reference describes how to configure Feast components: -* [Feast Core and Feast Online Serving](configuration-reference.md#2-feast-core-serving-and-job-controller) -* [Feast CLI and Feast Python SDK](configuration-reference.md#3-feast-cli-and-feast-python-sdk) -* [Feast Go and Feast Java SDK](configuration-reference.md#4-feast-java-and-go-sdk) +* [Feast Core and Feast Online Serving](./#2-feast-core-serving-and-job-controller) +* [Feast CLI and Feast Python SDK](./#3-feast-cli-and-feast-python-sdk) +* [Feast Go and Feast Java SDK](./#4-feast-java-and-go-sdk) ### 1. Feast Core and Feast Online Serving @@ -19,9 +19,9 @@ Available configuration properties for Feast Core and Feast Online Serving can b Configuration properties for Feast Core and Feast Online Serving are defined depending on Feast is deployed: -* [Docker Compose deployment](configuration-reference.md#docker-compose-deployment) - Feast is deployed with Docker Compose. -* [Kubernetes deployment](configuration-reference.md#kubernetes-deployment) - Feast is deployed with Kubernetes. -* [Direct Configuration](configuration-reference.md#direct-configuration) - Feast is built and run from source code. +* [Docker Compose deployment](./#docker-compose-deployment) - Feast is deployed with Docker Compose. +* [Kubernetes deployment](./#kubernetes-deployment) - Feast is deployed with Kubernetes. +* [Direct Configuration](./#direct-configuration) - Feast is built and run from source code. ### Docker Compose Deployment @@ -75,7 +75,7 @@ If Feast is built and running from source, configuration properties can be set d ### 2. Feast CLI and Feast Python SDK -Configuration options for both the [Feast CLI](../getting-started/connecting-to-feast-1/connecting-to-feast.md) and [Feast Python SDK](https://api.docs.feast.dev/python/) can be defined in the following locations, in order of precedence: +Configuration options for both the [Feast CLI](../../getting-started/connect-to-feast/connecting-to-feast.md) and [Feast Python SDK](https://api.docs.feast.dev/python/) can be defined in the following locations, in order of precedence: **1. Command line arguments or initialized arguments:** Passing parameters to the Feast CLI or instantiating the Feast Client object with specific parameters will take precedence above other parameters. diff --git a/docs/reference/metrics-reference.md b/docs/reference/metrics-reference/README.md similarity index 94% rename from docs/reference/metrics-reference.md rename to docs/reference/metrics-reference/README.md index 0a17629b7f..4ccd6cb9ce 100644 --- a/docs/reference/metrics-reference.md +++ b/docs/reference/metrics-reference/README.md @@ -2,11 +2,11 @@ Reference of the metrics that each Feast component exports: -* [Feast Core](metrics-reference.md#feast-core) -* [Feast Serving](metrics-reference.md#feast-serving) -* [Feast Ingestion Job](metrics-reference.md#feast-ingestion-job) +* [Feast Core](./#feast-core) +* [Feast Serving](./#feast-serving) +* [Feast Ingestion Job](./#feast-ingestion-job) -For how to configure Feast to export Metrics, see the [Metrics user guide.](../advanced/metrics.md) +For how to configure Feast to export Metrics, see the [Metrics user guide.](../../advanced/metrics.md) ## Feast Core @@ -44,8 +44,8 @@ Feast Serving exports the following metrics: | :--- | :--- | :--- | | `feast_serving_request_latency_seconds` | Feast Serving's latency in serving Requests in Seconds. | `method` | | `feast_serving_request_feature_count` | No. of requests retrieving a Feature from Feast Serving. | `project`, `feature_name` | -| `feast_serving_not_found_feature_count` | No. of requests retrieving a Feature has resulted in a [`NOT_FOUND` field status.](../user-guide/feature-retrieval.md#online-field-statuses) | `project`, `feature_name` | -| `feast_serving_stale_feature_count` | No. of requests retrieving a Feature resulted in a [`OUTSIDE_MAX_AGE` field status.](../user-guide/feature-retrieval.md#online-field-statuses) | `project`, `feature_name` | +| `feast_serving_not_found_feature_count` | No. of requests retrieving a Feature has resulted in a [`NOT_FOUND` field status.](../../user-guide/feature-retrieval.md#online-field-statuses) | `project`, `feature_name` | +| `feast_serving_stale_feature_count` | No. of requests retrieving a Feature resulted in a [`OUTSIDE_MAX_AGE` field status.](../../user-guide/feature-retrieval.md#online-field-statuses) | `project`, `feature_name` | | `feast_serving_grpc_request_count` | Total gRPC requests served. | `method` | **Metric Tags** @@ -61,7 +61,7 @@ Exported Feast Serving metrics may be filtered by the following tags/keys ## Feast Ingestion Job -Feast Ingestion computes both metrics an statistics on [data ingestion.](../user-guide/data-ingestion.md) Make sure you familar with data ingestion concepts before proceeding. +Feast Ingestion computes both metrics an statistics on [data ingestion.](../../user-guide/data-ingestion.md) Make sure you familar with data ingestion concepts before proceeding. {% hint style="info" %} For documentation on Feature value statistics computed by the Ingestion Job see [Statistics]() diff --git a/docs/roadmap.md b/docs/roadmap.md index ace85ff410..5ec26f85a1 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,32 +1,32 @@ # Roadmap -## Feast 0.8 +### Feast 0.8 [Discussion](https://github.com/feast-dev/feast/issues/1018) [Feast 0.8 RFC](https://docs.google.com/document/d/1snRxVb8ipWZjCiLlfkR4Oc28p7Fkv_UXjvxBFWjRBj4/edit#heading=h.yvkhw2cuvx5) -### **New Functionality** +#### **New Functionality** 1. Add support for AWS \(data sources and deployment\) 2. Add support for local deployment 3. Add support for Spark based ingestion 4. Add support for Spark based historical retrieval -### **Technical debt, refactoring, or housekeeping** +#### **Technical debt, refactoring, or housekeeping** 1. Move job management functionality to SDK 2. Remove Apache Beam based ingestion 3. Allow direct ingestion from batch sources that does not pass through stream 4. Remove Feast Historical Serving abstraction to allow direct access from Feast SDK to data sources for retrieval -## Feast 0.7 +### Feast 0.7 [Discussion](https://github.com/feast-dev/feast/issues/834) [GitHub Milestone](https://github.com/feast-dev/feast/milestone/4) -### **New Functionality** +#### **New Functionality** 1. Label based Ingestion Job selector for Job Controller [\#903](https://github.com/feast-dev/feast/pull/903) 2. Authentication Support for Java & Go SDKs [\#971](https://github.com/feast-dev/feast/pull/971) @@ -35,19 +35,19 @@ 5. Request Response Logging support via Fluentd [\#961](https://github.com/feast-dev/feast/pull/961) 6. Feast Core Rest Endpoints [\#878](https://github.com/feast-dev/feast/pull/878) -### **Technical debt, refactoring, or housekeeping** +#### **Technical debt, refactoring, or housekeeping** 1. Improved integration testing framework [\#886](https://github.com/feast-dev/feast/pull/886) 2. Rectify all flaky batch tests [\#953](https://github.com/feast-dev/feast/pull/953), [\#982](https://github.com/feast-dev/feast/pull/982) 3. Decouple job management from Feast Core [\#951](https://github.com/feast-dev/feast/pull/951) -## Feast 0.6 +### Feast 0.6 [Discussion](https://github.com/feast-dev/feast/issues/767) [GitHub Milestone](https://github.com/feast-dev/feast/milestone/3) -### New functionality +#### New functionality 1. Batch statistics and validation [\#612](https://github.com/feast-dev/feast/pull/612) 2. Authentication and authorization [\#554](https://github.com/feast-dev/feast/pull/554) @@ -55,23 +55,23 @@ 4. Improved searching and filtering of features and entities 5. Python support for labels [\#663](https://github.com/feast-dev/feast/issues/663) -### Technical debt, refactoring, or housekeeping +#### Technical debt, refactoring, or housekeeping 1. Improved job life cycle management [\#761](https://github.com/feast-dev/feast/issues/761) 2. Compute and write metrics for rows prior to store writes [\#763](https://github.com/feast-dev/feast/pull/763) -## Feast 0.5 +### Feast 0.5 [Discussion](https://github.com/gojek/feast/issues/527) -### New functionality +#### New functionality 1. Streaming statistics and validation \(M1 from [Feature Validation RFC](https://docs.google.com/document/d/1TPmd7r4mniL9Y-V_glZaWNo5LMXLshEAUpYsohojZ-8/edit)\) 2. Support for Redis Clusters \([\#478](https://github.com/gojek/feast/issues/478), [\#502](https://github.com/gojek/feast/issues/502)\) 3. Add feature and feature set labels, i.e. key/value registry metadata \([\#463](https://github.com/gojek/feast/issues/463)\) 4. Job management API \([\#302](https://github.com/gojek/feast/issues/302)\) -### Technical debt, refactoring, or housekeeping +#### Technical debt, refactoring, or housekeeping 1. Clean up and document all configuration options \([\#525](https://github.com/gojek/feast/issues/525)\) 2. Externalize storage interfaces \([\#402](https://github.com/gojek/feast/issues/402)\) diff --git a/docs/why-feast.md b/docs/why-feast.md deleted file mode 100644 index d20709ddf8..0000000000 --- a/docs/why-feast.md +++ /dev/null @@ -1,32 +0,0 @@ -# Why Feast? - -## Lack of feature reuse - -**Problem:** The process of engineering features is one of the most time consuming activities in building an end-to-end ML system. Despite this, many teams continue to redevelop the same features from scratch for every new project. Often these features never leave the notebooks or pipelines they are built in. - -**Solution:** A centralized feature store allows organizations to build up a foundation of features that can be reused across projects. Teams are then able to utilize features developed by other teams, and as more features are added to the store it becomes easier and cheaper to build models. - -## Serving features is hard - -**Problem:** Serving up to date features at scale is hard. Raw data can come from a variety of sources, from data lakes, to event streams, to data warehouses, to simply flat files. Data scientists need the ability to produce massive datasets of features from this data in order to train their models offline. These models then need access to real-time feature data at low latency and high throughput when they are served in production. - -**Solution:** Feast is built to be able to ingest data from a variety of sources, supporting both streaming and batch sources. Once data is loaded into Feast as features, it becomes available through both a historical serving API as well as an real-time \(online serving\) API. These APIs allow data scientists and ML engineers to easily retrieve feature data for their development, training, or in production. Feast also comes with a Java, Go, and Python SDK to make this experience easy. - -## **Models need point-in-time correctness** - -**Problem:** Most data sources are not built with ML use cases in mind and by extension don't provide point-in-time correct lookups of feature data. One of the reasons why features are often re-engineered is because ML practitioners need to ensure that their models are trained on a dataset that accurately models the state of the world when the model runs in production. - -**Solution:** Feast allows end users to create point-in-time correct datasets across multiple entities. Feast ensures that there is no data leakage, that cross feature set joins are valid, and that models are not fed expired data. - -## Definitions of features vary - -**Problem:** Without a centralized definition of what a feature means, teams inevitably define and document features differently, which further reduces the ability of teams to understand, trust, and reuse features. - -**Solution:** Feast becomes the single source of truth for all feature data for all models within an organization. Teams are able to capture documentation, metadata and metrics about features. This allows teams to communicate clearly about features, test features data, and determine if a feature is useful for a particular model. - -## **Inconsistency between training and serving** - -**Problem:** Training requires access to historical data, whereas models that serve predictions need the latest values. Inconsistencies arise when data is siloed into many independent systems requiring separate tooling. Often teams are using Python for creating batch features offline, but these features are redeveloped with different libraries and languages when moving to serving or streaming systems. - -**Solution:** Feast provides consistency by managing and unifying the ingestion of data from batch and streaming sources into both the feature warehouse and feature serving stores. Feast becomes the bridge between your model and your data, both for training and serving. This ensures that there is a consistency in the feature data that your model receives. -