diff --git a/spiceaidocs/config.toml b/spiceaidocs/config.toml index 91ee42cd..29f01576 100644 --- a/spiceaidocs/config.toml +++ b/spiceaidocs/config.toml @@ -59,10 +59,10 @@ anchor = "smart" # Everything below this are Site Params -[[menu.left]] - name = "API" - weight = 10 - url = "api" +# [[menu.left]] +# name = "API" +# weight = 10 +# url = "api" [[menu.left]] name = "CLI" weight = 10 diff --git a/spiceaidocs/content/en/_index.md b/spiceaidocs/content/en/_index.md index 622cc667..7a1ffb32 100644 --- a/spiceaidocs/content/en/_index.md +++ b/spiceaidocs/content/en/_index.md @@ -86,30 +86,9 @@ Thank you for sharing this journey with us. 🙏 -
-
-
Quickstarts
-

Quickstart tutorials to get started with Spice.ai quickly. 🚀

- -
-
-
-
-
Samples
-

Learn about Spice.ai with in-depth samples.

- -
-

-
-
-
API
-

Documentation on the Spice.ai API.

- -
-
CLI
diff --git a/spiceaidocs/content/en/api/_index.md b/spiceaidocs/content/en/api/_index.md deleted file mode 100644 index 5da7e9b0..00000000 --- a/spiceaidocs/content/en/api/_index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: swagger -title: "Spice.ai API documentation" -linkTitle: "API" -weight: 60 -description: "OpenAPI documentation for the Spice.ai API" ---- - -{{< swaggerui src="/openapi/spiceai.yaml" >}} diff --git a/spiceaidocs/content/en/concepts/_index.md b/spiceaidocs/content/en/concepts/_index.md index 7ce486d7..ae3ca221 100644 --- a/spiceaidocs/content/en/concepts/_index.md +++ b/spiceaidocs/content/en/concepts/_index.md @@ -32,7 +32,7 @@ The `spice` command line tool for interacting with the Spice.ai runtime during d ## Spice Runtime -The Spice.ai core runtime which runs as the daemon `spiced` and includes the AI and Data engines. The runtime also serves the [Spice.ai API]({{}}). +The Spice core runtime which runs as the daemon `spiced` and includes the AI and Data engines. ## [spicerack.org](https://spicerack.org) diff --git a/spiceaidocs/content/en/concepts/dataspaces/_index.md b/spiceaidocs/content/en/concepts/dataspaces/_index.md deleted file mode 100644 index 11860c68..00000000 --- a/spiceaidocs/content/en/concepts/dataspaces/_index.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -type: docs -title: "Dataspaces" -linkTitle: "Dataspaces" -weight: 20 -description: "Reference documentation for Spice.ai Dataspaces" ---- - -A **dataspace** is a logical grouping of data with definitions of how that data should be loaded and processed, usually from a single source. - -A combination of its [data source](https://github.com/spiceai/data-components-contrib/tree/trunk/dataprocessors) and its [name](https://github.com/spiceai/data-components-contrib/tree/trunk/dataprocessors) identifies it, for example, `nasdaq/msft` or `twitter/tweets`. - -The [dataspaces node](https://github.com/spiceai/data-components-contrib/tree/trunk/dataprocessors) of the [Spicepod manifest](https://docs.spiceai.org/reference/pod/) may hold one or more dataspace definitions. The runtime will merge data from all pod dataspaces into observations. - -Each dataspace encapsulates definitions for its core data primitives, such as [measurements](https://docs.spiceai.org/reference/pod/#dataspacesmeasurements) (numerical data), [categories](https://docs.spiceai.org/reference/pod/#dataspacescategories) (string-based categorical data), and [tags](https://docs.spiceai.org/reference/pod/#dataspacestags) (string-based tags). It also includes configuration for data connector and data processor components to load and process data into those primitives. - -While measurements and categories are scoped to the dataspace namespace, tags are aggregated with tags from other dataspaces to the pod scope. - -Dataspaces may load data through a [data connector](https://docs.spiceai.org/concepts/#data-connector) and [processor](https://docs.spiceai.org/concepts/#data-processor) or through the POST [/pods/{pod}/observations]({{}}) API. The API accepts JSON or CSV data and uses the appropriate [JSON](https://github.com/spiceai/data-components-contrib/blob/trunk/dataprocessors/json/README.md) or [CSV](https://github.com/spiceai/data-components-contrib/tree/trunk/dataprocessors/csv) data processor. - -Data connectors and processors are community-maintained components for streaming and processing time-series data. More information can be found in the [data-components-contrib](https://github.com/spiceai/data-components-contrib) repository. - -### Reference - -- List of [supported data connectors](https://github.com/spiceai/data-components-contrib/tree/trunk/dataconnectors/README.md) -- List of [supported data processors](https://github.com/spiceai/data-components-contrib/blob/trunk/dataprocessors/README.md) -- [API Reference]({{}}) diff --git a/spiceaidocs/content/en/concepts/interpretations/_index.md b/spiceaidocs/content/en/concepts/interpretations/_index.md deleted file mode 100644 index cfde1505..00000000 --- a/spiceaidocs/content/en/concepts/interpretations/_index.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -type: docs -title: "Interpretations" -linkTitle: "Interpretations" -weight: 15 -description: "Reference documentation for Spice.ai Intepretations" ---- - -- [API Reference]({{}}) - -Interpretations provide a way to define meaning for a time range within a pod period. For example, for a stock trading pod with a period of Monday to Friday, an interpretation of that time range could be, from Tuesday to Wednesday is a good time to buy. - -An example interpretation defined in JSON is as follows: - -```json -{ - "start": 1605333600, - "end": 1605333601, - "name": "e2e-test-interpretation", - "actions": [ - "small_buy", - ], - "tags": [ - "mytag", - ] - }, -``` - -The interpretation is defined as a time range from `start` to `end`, with a `name` and a list of `actions` and `tags`. - -Interpretations can be used to provide hints to the reward function on how to reward a time step. In the above example, when the training reaches Tuesday, the reward function author might choose to reward buys even higher based on that expert input. - -When the action specific reward function is called, if there is an interpretation in that time range, it will be provided to the reward function in `[state]_interpretations`. E.g. if an interpretation overlapped with new state then `next_state_interpretations` would contain a list of the overlapping interpretations. - -Comparing Spice.ai recommendations to interpretations is also one way of testing Spice.ai recommendations against expected actions for input data. diff --git a/spiceaidocs/content/en/concepts/recommendations/_index.md b/spiceaidocs/content/en/concepts/recommendations/_index.md deleted file mode 100644 index f57a6b9f..00000000 --- a/spiceaidocs/content/en/concepts/recommendations/_index.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -type: docs -title: "Recommendations" -linkTitle: "Recommendations" -weight: 13 -description: "Reference documentation for Spice.ai Recommendations" ---- - -- [API Reference]({{}}) - -Recommendations are core to making intelligent apps that learn and adapt. - -Adaptive applications use Spice.ai Recommendations to inform their behavior and decisions on what actions to take during their operation. Each recommendation includes a confidence score, from zero to one. Applications should decide on the appropriate level of confidence for its given scenario as to act upon the recommendation or not. - -Valid recommendations (confidence scores greater than zero) are returned after the first training run for a pod is complete. - -If a time is not specified, the resulting recommendation query time will default to the time of the most recently ingested observation within the pod period. - -## API Format - -Get a recommendation using the latest ingested observation time: - -`GET http://localhost:8000/api/v0.1/pods/{pod}/recommendation` - -Get a recommendation for a specified time: - -`GET http://localhost:8000/api/v0.1/pods/{pod}/recommendation?time={unix_timestamp}` - -where `{unix_timestamp}` is specified in [Unix time](https://en.wikipedia.org/wiki/Unix_time) in seconds. See [Time]({{}}) for more information on how Spice.ai handles time. - -## Example - -For `quickstarts/trader` a valid request at a specific time is: - -`GET http://localhost:8000/api/v0.1/pods/trader/recommendation?time=1605729600` diff --git a/spiceaidocs/content/en/concepts/time/_index.md b/spiceaidocs/content/en/concepts/time/_index.md deleted file mode 100644 index b2e921ca..00000000 --- a/spiceaidocs/content/en/concepts/time/_index.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -type: docs -title: "Time" -linkTitle: "Time" -weight: 5 -description: "The concept of Time and Time-Series in Spice.ai" ---- - -Spice.ai is a time series AI platform, so time is one of Spice.ai's most fundamental and core concepts. - -## What is time series data? - -Time series data is a series of timestamped data points or events indexed in time order. In Spice.ai, these data points are called Observations. For example, temperature sensor readings on an interval or a daily stock price are examples of time series data. Time series data applies to many domains, including analytics, finance, health and biometrics, IoT and industrial, security, and application monitoring. To learn more, Wikipedia has a comprehensive article on [time series](https://en.wikipedia.org/wiki/Time_series), and InfluxData has published a very informative article, [What is time series data?](https://www.influxdata.com/what-is-time-series-data/) - -## Why time series AI? - -One definition of artificial intelligence proposed by [Hado van Hasselt from DeepMind](https://www.youtube.com/watch?v=TCCjZe0y4Qc&list=PLqYmG7hTraZDVH599EItlEWsUOsJbAodm) is "to be able to learn to make decisions to achieve goals." Using this definition can conclude that these decisions are made over time to achieve an application's goals. Thus, time and time-series data can be seen as fundamental concepts when developing an application's intelligence. - -## Time series in Spice.ai - -Spice.ai natively ingests, processes, and learns from time series data. There are three core time series concepts in Spice.ai: - -- A time series **Period** and its period **Epoch** -- Consecutive **intervals** or windows of time series observations -- The smallest **granularity** of time in the series -A visualization of these concepts over a timeline is below: - -spiceai-time - -Each of these concepts translates to developer configurable parameters in the Spicepod. For example, the Trader Sample uses these pod parameters in its manifest: - -```yaml -name: trader -params: - period: 30m - interval: 30s - granularity: 5s -``` - -If not provided in the manifest, Spicepods will default to a period of **3 days**, intervals of **1 min**, and granularity of **10 seconds**. The period epoch will default to a dynamic epoch of the current time minus the period. In this mode, the period becomes a sliding window over time. - -### Period - -The `period` defines the entire timespan the Spicepod will use for learning and decision-making. - -Thus the time series period has a start time of `epoch` and an end time of `epoch` + `period`. - -### Period Epoch - -The period `epoch` defines the beginning, or start, of the Spicepod's time series data. Spice.ai will discard data timestamped before the epoch time. - -The epoch defaults to the current time - `period`. - -### Interval - -The `interval` defines the interval or window of time the AI engine uses to learn. - -The first interval of data, from `epoch` to `epoch` + `interval` is considered pre-training or "warm-up" data. At least one interval's worth of data is required before the AI engine can learn and make a decision recommendation. - -The AI engine then trains on consecutive data intervals for the rest of the period to learn. - -### Granularity - -The `granularity` defines the smallest unit of time series data. The granularity can be conceptualized in practice as the time a decision is valid. For example, a decision to turn on or off an air conditioner would be valid for a `granularity` unit of time. - -Spice.ai will aggregate observations within a granularity timespan. This aggregation results in granularity-sized "ticks" or time-steps in the series. - -Each consecutive training interval advances by a unit of granularity over the entire period. Thus, the granularity cannot be larger than the interval. diff --git a/spiceaidocs/content/en/deep-learning-ai/_index.md b/spiceaidocs/content/en/deep-learning-ai/_index.md deleted file mode 100644 index 4abe2d44..00000000 --- a/spiceaidocs/content/en/deep-learning-ai/_index.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -type: docs -title: "Deep Learning AI" -linkTitle: "Deep Learning AI" -weight: 40 ---- - -The Spice.ai engine learns and provides recommendations to your application using a type of AI called deep reinforcement learning. - -Reinforcement learning (RL) is a general framework where agents learn to perform actions in an environment so as to maximize a reward according to a policy. In deep reinforcement learning, the policy is trained by a neural network based on a deep learning algorithm. - -The agent and environment continuously interact with each other. At each time step, the agent takes an action on the observation space based on its policy (aka brain), and receives a reward and the next observation from the environment. The goal is to improve the policy so as to maximize the sum of rewards (score). - -Spice.ai provides a standard interface that a deep learning algorithm can be implemented with. At launch Spice.ai supports two deep learning algorithms and more will be added over time. - -By default, Spice.ai will use [Deep Q-Learning]({{}}). To use a different algorithm, call `spice train` with the parameter `--learning-algorithm` set to one of the following values: - -| --learning-algorithm | Algorithm | -| -------------------- | ---------------------------------------------------------------- | -| dql | [Deep Q-Learning]({{}}) | -| vpg | [Vanilla Policy Gradient]({{}}) | -| sacd | [Soft Actor-Critic (Discrete)]({{}}) | - -**Example** - -```bash -spice train --learning-algorithm vpg -``` diff --git a/spiceaidocs/content/en/deep-learning-ai/dql.md b/spiceaidocs/content/en/deep-learning-ai/dql.md deleted file mode 100644 index b99e957d..00000000 --- a/spiceaidocs/content/en/deep-learning-ai/dql.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -type: docs -title: "Deep Q-Learning" -linkTitle: "Deep Q-Learning" -weight: 50 -description: Spice.ai implementation of the Deep Q Learning algorithm (DQL) ---- - -The DQN (Deep Q-Network) algorithm was developed by DeepMind in 2015. It was able to solve a wide range of Atari games (some to superhuman level) by combining reinforcement learning and deep neural networks at scale. The algorithm was developed by enhancing a classic RL algorithm called Q-Learning with deep neural networks and a technique called experience replay. - -Learn more about the Deep Q-Network algorithm at https://www.tensorflow.org/agents/tutorials/0_intro_rl diff --git a/spiceaidocs/content/en/deep-learning-ai/interface.md b/spiceaidocs/content/en/deep-learning-ai/interface.md deleted file mode 100644 index 18897b85..00000000 --- a/spiceaidocs/content/en/deep-learning-ai/interface.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -type: docs -title: "Interface" -linkTitle: "Interface" -weight: 30 -description: Learn how to add a new deep RL algorithm by implementing the Spice.ai interface ---- - -Implement a deep RL agent that is compatible with Spice.ai by implementing the following interface. This interface lives in code at: https://github.com/spiceai/spiceai/blob/trunk/ai/src/algorithms/agent_interface.py - -```python -""" -The interface that all Spice.ai compatible deep-RL agents should conform to. -""" - -class SpiceAIAgent(): - """ - Args: - state_shape: The shape of the observation state - action_size: How many actions our agent is able to take. - """ - - def __init__(self, state_shape, action_size): - self.state_shape = state_shape - self.action_size = action_size - - def act(self, state) -> Tuple[int, list]: - """ - Returns an action recommended by the algorithm's policy for the current state - - Args: - state: The observation state to act upon - - Returns: - (int): The action to take, as an integer that can index into the array of defined actions. - (list): The probabilities of each action. - """ - raise NotImplementedError() - - def add_experience(self, state, action, reward, next_state): - """ - Adds the experience of the reward from taking this action at that state - - Args: - state: The observation state when the action was taken - action: The action that was performed - reward: The reward that was received for taking that action - next_state: The observation state after the action was taken - """ - raise NotImplementedError() - - def learn(self): - """ - Updates the algorithm's policy based on its collected experiences so far. - """ - raise NotImplementedError() - - def save(self, model_path: str): - """ - Save the trained model to disk - - Args: - model_path: The path on disk to save the model parameters to - """ - raise NotImplementedError() - - def load(self, model_path: str) -> bool: - """ - Load a previously trained model from disk. Returns whether it was able to load the model. - - Args: - model_path: The path on disk to load the model parameters from - - Returns: - (bool) Whether the model was able to be loaded properly - """ - raise NotImplementedError() -``` diff --git a/spiceaidocs/content/en/deep-learning-ai/sac.md b/spiceaidocs/content/en/deep-learning-ai/sac.md deleted file mode 100644 index a728394d..00000000 --- a/spiceaidocs/content/en/deep-learning-ai/sac.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -type: docs -title: "Soft Actor-Critic" -linkTitle: "Soft Actor-Critic" -weight: 50 -description: Spice.ai implementation of the Soft Actor-Critic algorithm (SAC) ---- - -The SAC (Soft Actor-Critic) algorithm was developed in 2018. It is a off-policy, model-free reinforcement learning algorithm that aims not only at maximizing the reward but also the entropy (acting as randomly as possible). The entropy maximization helps exploring possibilities and trying actions that seems to be equally rewarding. - -The Spice.ai implementation of Soft Actor-Critic has been modified to work for discrete action sets. - -Berkeley AI Research blog: https://bair.berkeley.edu/blog/2018/12/14/sac/ -Arxiv paper: https://arxiv.org/abs/1801.01290 diff --git a/spiceaidocs/content/en/deep-learning-ai/vpg.md b/spiceaidocs/content/en/deep-learning-ai/vpg.md deleted file mode 100644 index bfc800fc..00000000 --- a/spiceaidocs/content/en/deep-learning-ai/vpg.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -type: docs -title: "Vanilla Policy Gradient" -linkTitle: "Vanilla Policy Gradient" -weight: 40 -description: Spice.ai implementation of the Vanilla Policy Gradient algorithm (VPG) ---- - -The key idea underlying policy gradients is to push up the probabilities of -actions that lead to higher return, and push down the probabilities of actions -that lead to lower return, until you arrive at the optimal policy. - -## Exploration vs. Exploitation - -VPG trains a stochastic policy in an on-policy way. This means that it explores -by sampling actions according to the latest version of its stochastic policy. -The amount of randomness in action selection depends on both initial conditions -and the training procedure. Over the course of training, the policy typically -becomes progressively less random, as the update rule encourages it to exploit -rewards that it has already found. This may cause the policy to get trapped in local optima. - -Learn more about the Vanilla Policy Gradient algorithm at https://spinningup.openai.com/en/latest/algorithms/vpg.html diff --git a/spiceaidocs/content/en/getting-started/_index.md b/spiceaidocs/content/en/getting-started/_index.md index 27280c50..6bd67702 100644 --- a/spiceaidocs/content/en/getting-started/_index.md +++ b/spiceaidocs/content/en/getting-started/_index.md @@ -14,10 +14,9 @@ Before following this guide it is first recommended to review the Spice.ai [core ### Follow these steps to get started with Spice.ai. 1. Install Spice.ai -1. Create your first Spice.ai pod and train it -1. Observe your pod training -1. Get a recommendation from your pod -1. Explore Spice.ai quickstarts - + + + + {{< button text="First step: Install Spice.ai >>" page="install-spiceai" >}} diff --git a/spiceaidocs/content/en/quickstarts/_index.md b/spiceaidocs/content/en/quickstarts/_index.md deleted file mode 100644 index 2c7c8b12..00000000 --- a/spiceaidocs/content/en/quickstarts/_index.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -type: docs -title: "Quickstarts" -linkTitle: "Quickstarts" -weight: 21 -description: "Quickstart tutorials to get started with Spice.ai quickly 🚀" -no_list: true ---- - -## Spice.ai Quickstart Tutorials - -These simple tutorials use static data to help you get started with Spice.ai quickly. For examples of Spicepods that use streaming data, see [Samples]({{}}). - -- [ServerOps](https://github.com/spiceai/quickstarts/tree/trunk/serverops/README.md) - A CPU metrics based server operations app -- [Trader](https://github.com/spiceai/quickstarts/tree/trunk/trader/README.md) - A basic Bitcoin trading bot -- [Tweet Recommendation](https://github.com/spiceai/quickstarts/tree/trunk/tweet-recommendation/README.md) - A Twitter action recommendation app diff --git a/spiceaidocs/content/en/samples/_index.md b/spiceaidocs/content/en/samples/_index.md deleted file mode 100644 index 77c3fc3b..00000000 --- a/spiceaidocs/content/en/samples/_index.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -type: docs -title: "Samples" -linkTitle: "Samples" -weight: 22 -description: "Learn about Spice.ai with in-depth samples" -no_list: true ---- - -## Learn about Spice.ai with in-depth samples - -These tutorials are more in-depth than the [Quickstarts]({{}}) and use live streaming data for learning. - -### Apps that learn and adapt - -- [ServerOps](https://github.com/spiceai/samples/tree/trunk/serverops/README.md) - Run server maintainance during periods of low load. -- [Gardener](https://github.com/spiceai/samples/tree/trunk/gardener/README.md) - Intelligently water a simulated garden. -- [Trader](https://github.com/spiceai/samples/tree/trunk/trader/README.md) - Bitcoin trading from Coinbase Pro exchange data. - -### Deploy Spice.ai - -- [Kubernetes](https://github.com/spiceai/samples/tree/trunk/kubernetes/README.md) - Use Spice.ai in your Kubernetes cluster. diff --git a/spiceaidocs/content/en/training/_index.md b/spiceaidocs/content/en/training/_index.md deleted file mode 100644 index e53ca105..00000000 --- a/spiceaidocs/content/en/training/_index.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -type: docs -title: "Spice.ai Training Docs" -linkTitle: "Training" -weight: 30 -description: "Training documentation." ----