Skip to content

Commit

Permalink
Update website with code block copy button and nord theme
Browse files Browse the repository at this point in the history
  • Loading branch information
lbeckman314 committed Oct 11, 2024
1 parent f3edfec commit 3b2ecca
Show file tree
Hide file tree
Showing 25 changed files with 399 additions and 217 deletions.
2 changes: 1 addition & 1 deletion website/content/docs/compute/aws-batch.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ AmazonS3:
### Start the server
```
```sh
funnel server run --config /path/to/config.yaml
```

Expand Down
4 changes: 2 additions & 2 deletions website/content/docs/compute/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@ these services without using nodes or the scheduler.
### Usage

Nodes are available via the `funnel node` command. To start a node, run
```
```sh
funnel node run --config node.config.yml
```

To activate the Funnel scheduler, use the `manual` backend in the config.

The available scheduler and node config:
```
```yaml
# Activate the Funnel scheduler.
Compute: manual

Expand Down
193 changes: 6 additions & 187 deletions website/content/docs/compute/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,37 +10,12 @@ menu:

This guide will take you through the process of setting up Funnel as a kubernetes service.

Kuberenetes Resources:
- [Service](https://kubernetes.io/docs/concepts/services-networking/service/)
- [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)
- [ConfigMap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/)
- [Roles and RoleBindings](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#default-roles-and-role-bindings)
- [Job](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/)

Additional Funnel deployment resources can be found here: https://github.com/ohsu-comp-bio/funnel/tree/master/deployments/kubernetes

#### Create a Service:

*funnel-service.yml*

```yaml
apiVersion: v1
kind: Service
metadata:
name: funnel
spec:
selector:
app: funnel
ports:
- name: http
protocol: TCP
port: 8000
targetPort: 8000
- name: rpc
protocol: TCP
port: 9090
targetPort: 9090

{{< read-file "static/funnel-config-examples/kubernetes/funnel-service.yml" >}}
```

Deploy it:
Expand All @@ -62,65 +37,7 @@ Use this value to configure the server hostname of the worker config.
*funnel-server-config.yml*

```yaml
Database: boltdb

Compute: kubernetes

Logger:
Level: debug

Kubernetes:
# The executor used to execute tasks. Available executors: docker, kubernetes
Executor: "kubernetes"
DisableJobCleanup: false
DisableReconciler: false
ReconcileRate: 5m
Namespace: default
Template: |
apiVersion: batch/v1
kind: Job
metadata:
## DO NOT CHANGE NAME
name: {{.TaskId}}
namespace: {{.Namespace}}
spec:
backoffLimit: 0
completions: 1
template:
spec:
restartPolicy: Never
serviceAccountName: funnel-sa
containers:
- name: {{printf "funnel-worker-%s" .TaskId}}
image: quay.io/ohsu-comp-bio/funnel:latest
imagePullPolicy: IfNotPresent
args:
- "worker"
- "run"
- "--config"
- "/etc/config/funnel-worker-config.yml"
- "--taskID"
- {{.TaskId}}
resources:
requests:
cpu: {{if ne .Cpus 0 -}}{{.Cpus}}{{ else }}{{"100m"}}{{end}}
memory: {{if ne .RamGb 0.0 -}}{{printf "%.0fG" .RamGb}}{{else}}{{"16M"}}{{end}}
ephemeral-storage: {{if ne .DiskGb 0.0 -}}{{printf "%.0fG" .DiskGb}}{{else}}{{"100M"}}{{end}}
volumeMounts:
- name: {{printf "funnel-storage-%s" .TaskId}}
mountPath: {{printf "/opt/funnel/funnel-work-dir/%s" .TaskId}}
- name: config-volume
mountPath: /etc/config
securityContext:
privileged: true
volumes:
- name: {{printf "funnel-storage-%s" .TaskId}}
emptyDir: {}
- name: config-volume
configMap:
name: funnel-config
{{< read-file "static/funnel-config-examples/kubernetes/funnel-server-config.yml" >}}
```

We recommend setting `DisableJobCleanup` to `true` for debugging - otherwise failed jobs will be cleanup up.
Expand All @@ -130,31 +47,7 @@ We recommend setting `DisableJobCleanup` to `true` for debugging - otherwise fai
***Remember to modify the template below to have the actual server hostname.***

```yaml
Database: boltdb
BoltDB:
Path: /opt/funnel/funnel-work-dir/funnel.bolt.db
Compute: kubernetes
Kubernetes:
# The executor used to execute tasks. Available executors: docker, kubernetes
Executor: "kubernetes"
Logger:
Level: debug
RPCClient:
MaxRetries: 3
Timeout: 30s
EventWriters:
- rpc
- log
Server:
HostName: < funnel service clusterIP >
RPCPort: 9090
{{< read-file "static/funnel-config-examples/kubernetes/funnel-worker-config.yml" >}}
```

#### Create a ConfigMap
Expand All @@ -170,41 +63,13 @@ Define a Role and RoleBinding:
*role.yml*

```yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: funnel-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
- apiGroups: ["batch", "extensions"]
resources: ["jobs"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
{{< read-file "static/funnel-config-examples/kubernetes/role.yml" >}}
```

*role_binding.yml*

```yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: funnel-rolebinding
namespace: default
subjects:
- kind: ServiceAccount
name: funnel-sa
roleRef:
kind: Role
name: funnel-role
apiGroup: rbac.authorization.k8s.io
{{< read-file "static/funnel-config-examples/kubernetes/role_binding.yml" >}}
```

Create the service account, role and role binding:
Expand All @@ -220,53 +85,7 @@ kubectl create -f role_binding.yml
*funnel-deployment.yml*

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: funnel
labels:
app: funnel
spec:
replicas: 1
selector:
matchLabels:
app: funnel
template:
metadata:
labels:
app: funnel
spec:
serviceAccountName: funnel-sa
containers:
- name: funnel
image: quay.io/ohsu-comp-bio/funnel:latest
imagePullPolicy: IfNotPresent
command:
- 'funnel'
- 'server'
- 'run'
- '--config'
- '/etc/config/funnel-server-config.yml'
resources:
requests:
cpu: 2
memory: 4G
ephemeral-storage: 25G # needed since we are using boltdb
volumeMounts:
- name: funnel-deployment-storage
mountPath: /opt/funnel/funnel-work-dir
- name: config-volume
mountPath: /etc/config
ports:
- containerPort: 8000
- containerPort: 9090
volumes:
- name: funnel-deployment-storage
emptyDir: {}
- name: config-volume
configMap:
name: funnel-config
{{< read-file "static/funnel-config-examples/kubernetes/funnel-deployment.yml" >}}
```

Deploy it:
Expand Down
2 changes: 1 addition & 1 deletion website/content/docs/databases/boltdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ and scheduler data. This is great for development and a simple server without
external dependencies, but it doesn't scale well to larger clusters.

Available config:
```
```yaml
Database: boltdb

BoltDB:
Expand Down
2 changes: 1 addition & 1 deletion website/content/docs/databases/datastore.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ such as stdout/err logs).
Funnel will, by default, try to will try to automatically load credentials from the environment. Alternatively, you may explicitly set the credentials in the config.

Config:
```
```yaml
Database: datastore

Datastore:
Expand Down
2 changes: 1 addition & 1 deletion website/content/docs/databases/dynamodb.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Funnel supports storing task data in DynamoDB. Storing scheduler data is not sup
Funnel will, by default, try to will try to automatically load credentials from the environment. Alternatively, you may explicitly set the credentials in the config.

Available Config:
```
```yaml
Database: dynamodb

DynamoDB:
Expand Down
2 changes: 1 addition & 1 deletion website/content/docs/databases/elasticsearch.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ menu:
Funnel supports storing tasks and scheduler data in Elasticsearch.

Config:
```
```yaml
Database: elastic

Elastic:
Expand Down
2 changes: 1 addition & 1 deletion website/content/docs/databases/mongodb.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ menu:
Funnel supports storing tasks and scheduler data in MongoDB.

Config:
```
```yaml
Database: mongodb

MongoDB:
Expand Down
2 changes: 1 addition & 1 deletion website/content/docs/events/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ menu:
Funnel supports writing task events to a Kafka topic. To use this, add an event
writer to the config:

```
```yaml
EventWriters:
- kafka
- log
Expand Down
4 changes: 2 additions & 2 deletions website/content/docs/integrations/py-tes.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@ menu:

Available on [PyPI](https://pypi.org/project/py-tes/).

```
```sh
pip install py-tes
```

### Example Python Script

```
```py
import tes

task = tes.Task(
Expand Down
4 changes: 2 additions & 2 deletions website/content/docs/storage/ftp.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@ Currently authentication credentials are take from the URL, e.g. `ftp://username
The FTP storage client is enabled by default, but may be explicitly disabled in the
worker config:

```
```yaml
FTPStorage:
Disabled: false
```
### Example task
```
```json
{
"name": "Hello world",
"inputs": [{
Expand Down
4 changes: 2 additions & 2 deletions website/content/docs/storage/google-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,15 @@ The Google storage client is enabled by default, and will try to automatically
load credentials from the environment. Alternatively, you
may explicitly set the credentials in the worker config:

```
```yaml
GoogleStorage:
Disabled: false
# Path to account credentials file.
AccountFile: ""
```
### Example task
```
```json
{
"name": "Hello world",
"inputs": [{
Expand Down
4 changes: 2 additions & 2 deletions website/content/docs/storage/http.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,15 @@ providers exposed using presigned URLs.
The HTTP storage client is enabled by default, but may be explicitly disabled in the
worker config:

```
```yaml
HTTPStorage:
Disabled: false
# Timeout for http(s) GET requests.
Timeout: 30s
```
### Example task
```
```json
{
"name": "Hello world",
"inputs": [{
Expand Down
2 changes: 1 addition & 1 deletion website/content/docs/storage/local.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Funnel limits which directories may be accessed, by default only allowing direct
under the current working directory of the Funnel worker.

Config:
```
```yaml
LocalStorage:
# Whitelist of local directory paths which Funnel is allowed to access.
AllowedDirs:
Expand Down
Loading

0 comments on commit 3b2ecca

Please sign in to comment.