diff --git a/website/config.toml b/website/config.toml index 52b84915a..434204fb4 100644 --- a/website/config.toml +++ b/website/config.toml @@ -3,6 +3,8 @@ languageCode = "en-us" title = "Dkron - Distributed job scheduling system" theme = "docdock" MetaDataFormat = "yaml" +pygmentsCodeFences = true +pygmentsStyle = "monokailight" [params] editURL = "https://github.com/victorcoder/dkron/edit/master/website/content/" diff --git a/website/content/api/_index.md b/website/content/api/_index.md index 6f8925bf2..03ca9b0f2 100644 --- a/website/content/api/_index.md +++ b/website/content/api/_index.md @@ -1,6 +1,6 @@ --- title: "API" -date: 2017-06-30T14:58:54+02:00 +weight: 100 --- diff --git a/website/content/basics/_index.md b/website/content/basics/_index.md index 74fcb739e..7f024a596 100644 --- a/website/content/basics/_index.md +++ b/website/content/basics/_index.md @@ -1,40 +1,7 @@ --- title: "Basics" -date: 2017-06-30T14:58:54+02:00 -draft: false +weight: 20 icon: "1. " --- -## Dkron - Distributed, fault tolerant job scheduling system - -Welcome to the Dkron documentation! This is the reference guide on how to use Dkron. If you want a getting started guide refer to the [getting started guide](getting-started). - -## What is Dkron - -Dkron is a distributed system to run scheduled jobs against a server or a group of servers of any size. One of the machines is the leader and the others will be followers. If the leader fails or becomes unreachable, any other one will take over and reschedule all jobs to keep the system healthy. - -In case the old leader becomes alive again, it'll become a follower. - -Dkron is a distributed cron drop-in replacement, easy to setup and fault tolerant with focus in: - -- Easy: Easy to use with a great UI -- Reliable: Completely fault tolerant -- High scalable: Able to handle high volumes of scheduled jobs and thousands of nodes - -Dkron is written in Go and leverage the power of distributed key value stores and [Serf](https://www.serfdom.io/) for providing fault tolerance, reliability and scalability while keeping simple and easily installable. - -Dkron is inspired by the google whitepaper [Reliable Cron across the Planet](https://queue.acm.org/detail.cfm?id=2745840) - -Dkron runs on Linux, OSX and Windows. It can be used to run scheduled commands on a server cluster using any combination of servers for each job. It has no single points of failure due to the use of the fault tolerant distributed databases and can work at large scale thanks to the efficient and lightweight gossip protocol. - -Dkron uses the efficient and lightweight [gossip protocol](https://www.serfdom.io/docs/internals/gossip.html) underneath to communicate with nodes. Failure notification and task handling are run efficiently across an entire cluster of any size. - -## Web UI - - -## Dkron design - -Dkron is designed to solve one problem well, executing commands in given intervals. Following the unix philosophy of doing one thing and doing it well (like the battle-tested cron) but with the given addition of being designed for the cloud era, removing single points of failure in environments where scheduled jobs are needed to be run in multiple servers. - +{{% children style="h2" depth="3" description="true" %}} diff --git a/website/content/basics/configuration.md b/website/content/basics/configuration.md index c78a3141f..dcc3fcd37 100644 --- a/website/content/basics/configuration.md +++ b/website/content/basics/configuration.md @@ -1,7 +1,6 @@ --- title: Configuration -wight: 2 -toc: false +wight: 20 --- Settings for dkron can be specified in three ways: Using a `config/dkron.json` config file, using env variables starting with `DKRON_` or using command line arguments. diff --git a/website/content/basics/getting-started.md b/website/content/basics/getting-started.md index 9ea79156f..378240c7f 100644 --- a/website/content/basics/getting-started.md +++ b/website/content/basics/getting-started.md @@ -1,10 +1,8 @@ --- title: Getting started -weight: 1 +weight: 30 --- -Welcome to the intro guide to dkron! This will explain how to setup dkron, how easy is to use it, what problems could it help you to solve, etc. - ## Introduction Dkron nodes can work in two modes, agents or servers. @@ -23,33 +21,28 @@ You can choose whether a job is run on a node or nodes by specifying tags and a All the execution responses will be gathered by the scheduler and stored in the database. -## Requirements +## Backend stores Dkron relies on the key-value store for data storage, an instance of the distributed store can be run in the same machines as Dkron or connect it to an already existing cluster. -It is compatible with etcd, Consul, Zookeeper and Redis data stores. For instructions on how to install and configure any one of these systems refer to their official sites: +{{% notice note %}} +By default dkron will start with a file based, embedded KV store called BoltDB, it is functional for a single node demo but does not offers clustering or HA. +{{% /notice %}} + +It is compatible with etcd, Consul, Zookeeper, BoltDB and partially with Redis. For instructions on how to install and configure any one of these systems refer to their official sites: - [etcd](https://coreos.com/etcd/docs/latest/) - [Consul](https://consul.io/intro/getting-started/install.html) - [ZooKeeper](https://zookeeper.apache.org/doc/r3.3.3/zookeeperStarted.html) - [Redis](https://redis.io/topics/quickstart) -## Installation - -### Recommended method - -APT repository: `deb [trusted=yes] https://apt.fury.io/victorcoder/ /` - -Unstable release: `sudo apt-get install dkron-unstable` -Stable release: `sudo apt-get install dkron` - -### Other methods +## Configuration -Simply download the packaged archive for your platform from the [downloads page](https://github.com/victorcoder/dkron/releases), extract the package to a shared location in your drive, like `/opt/local` and run it from there. +See the [installation](/basics/installation). ## Configuration -See the [configuration section](/basics/configuration). +See the [configuration](/basics/configuration). ## Usage @@ -59,9 +52,11 @@ By default Dkron uses the following ports: - `8080` for HTTP for the API and Dashboard - `6868` for RPC comunication between agents. -**Be sure you have opened this ports (or the ones that you configured) in your firewall or AWS security groups.** +{{% notice note %}} +Be sure you have opened this ports (or the ones that you configured) in your firewall or AWS security groups. +{{% /notice %}} -By default dkron will try to use a local etcd server running in the same machine and in the default port. A different store can be specified setting `backend` and `backend-machines` flag in the config file, env variables or as a command line flag. +By default dkron will use the embedded BoltDB KV store. A different store can be specified setting `backend` and `backend-machines` flag in the config file, env variables or as a command line flag. To start a Dkron server instance: @@ -72,7 +67,7 @@ dkron agent --server Time to add the first job: {{% notice note %}} -This job will only run in just one dkron_server node due to the node count in the tag. Refer to the [target node spec](/usage/target-nodes-spec) for details. +This job will only run in just one `dkron_server` node due to the node count in the tag. Refer to the [target node spec](/usage/target-nodes-spec) for details. {{% /notice %}} ```bash @@ -93,3 +88,7 @@ curl localhost:8080/v1/jobs -XPOST -d '{ } }` ``` + +That's it! + +#### To start configuring an HA installation of Dkron follow the [clustering guide](/usage/clustering) diff --git a/website/content/basics/installation.md b/website/content/basics/installation.md new file mode 100644 index 000000000..b7e32827b --- /dev/null +++ b/website/content/basics/installation.md @@ -0,0 +1,77 @@ + +--- +title: Installation +weight: 10 +--- + +## Running the binary + +Download the packaged archive for your platform from the [downloads page](https://github.com/victorcoder/dkron/releases) and extract the package to a shared location in your drive, like /opt/local/bin. + +Run Dkron with default setting: `dkron agent --server` + +Navigate to http://localhost:8080 + + +{{% notice note %}} +By default dkron will start with a file based, embedded KV store called BoltDB, it is functional for a single node demo but does not offers clustering or HA. +{{% /notice %}} + +## Installing the package + +### Debian repo + +APT repository: +``` +deb [trusted=yes] https://apt.fury.io/victorcoder/ / +``` + +Then install: `sudo apt-get install dkron` + +### YUM repo + +YUM repository: + +``` +[dkron] +name=Dkron Pro Private Repo +baseurl=https://yum.fury.io/victorcoder/ +enabled=1 +gpgcheck=0 +``` + +Then install: `sudo apt-get install dkron` + +This will start Dkron as a system service and the place example configuration file under `/etc/dkron/dkron.yml` + +## Running in Docker + +Dkron provides an official Docker image vi Dockerhub that can be used for deployment on any system running Docker. + +### Launching Dkron on a new container + +Here’s a quick one-liner to get you off the ground (please note, we recommend further configuration for production deployments below): + +``` +docker run -d -p 8080:8080 --name dkron dkron/dkron agent --server +``` + +This will launch a Dkron server on port 8080 by default. You can use docker logs -f dkron to follow the rest of the initialization progress. Once the Dkron startup completes you can access the app at localhost:8080 + +Since Docker containers have their own ports and we just map them to the system ports as needed it’s easy to move Dkron onto a different system port if you wish. For example running Dkron on port 12345: + +``` +docker run -d -p 12345:8080 --name dkron dkron/dkron +``` + +### Mounting a mapped file storage volume + +In its default configuration Dkron uses the local filesystem to run a BoltDB embedded database to store its own application data. The end result is that your Dkron application data will be on disk inside your container and lost if you ever remove the container. + +To persist your data outside of the container and make it available for use between container launches we can mount a local file path inside our container. + +``` +docker run -d -p 8080:8080 -v ~/dkron.db:/dkron.db --name dkron dkron/dkron agent --server +``` + +Now when you launch your container we are mounting that file from our local filesystem into the container. diff --git a/website/content/intro/_index.md b/website/content/intro/_index.md new file mode 100644 index 000000000..c6a42e6de --- /dev/null +++ b/website/content/intro/_index.md @@ -0,0 +1,38 @@ +--- +title: "Intro" +weight: 80 +icon: "1. " +--- + +## Dkron - Distributed, fault tolerant job scheduling system + +Welcome to the Dkron documentation! This is the reference guide on how to use Dkron. If you want a getting started guide refer to the [getting started guide](getting-started). + +## What is Dkron + +Dkron is a distributed system to run scheduled jobs against a server or a group of servers of any size. One of the machines is the leader and the others will be followers. If the leader fails or becomes unreachable, any other one will take over and reschedule all jobs to keep the system healthy. + +In case the old leader becomes alive again, it'll become a follower. + +Dkron is a distributed cron drop-in replacement, easy to setup and fault tolerant with focus in: + +- Easy: Easy to use with a great UI +- Reliable: Completely fault tolerant +- High scalable: Able to handle high volumes of scheduled jobs and thousands of nodes + +Dkron is written in Go and leverage the power of distributed key value stores and [Serf](https://www.serfdom.io/) for providing fault tolerance, reliability and scalability while keeping simple and easily installable. + +Dkron is inspired by the google whitepaper [Reliable Cron across the Planet](https://queue.acm.org/detail.cfm?id=2745840) + +Dkron runs on Linux, OSX and Windows. It can be used to run scheduled commands on a server cluster using any combination of servers for each job. It has no single points of failure due to the use of the fault tolerant distributed databases and can work at large scale thanks to the efficient and lightweight gossip protocol. + +Dkron uses the efficient and lightweight [gossip protocol](https://www.serfdom.io/docs/internals/gossip.html) underneath to communicate with nodes. Failure notification and task handling are run efficiently across an entire cluster of any size. + +## Web UI + + +## Dkron design + +Dkron is designed to solve one problem well, executing commands in given intervals. Following the unix philosophy of doing one thing and doing it well (like the battle-tested cron) but with the given addition of being designed for the cloud era, removing single points of failure in environments where scheduled jobs are needed to be run in multiple servers. diff --git a/website/content/dkron_vs_other_software.md b/website/content/intro/dkron_vs_other_software.md similarity index 100% rename from website/content/dkron_vs_other_software.md rename to website/content/intro/dkron_vs_other_software.md diff --git a/website/content/license.md b/website/content/intro/license.md similarity index 100% rename from website/content/license.md rename to website/content/intro/license.md diff --git a/website/content/operations/_index.md b/website/content/operations/_index.md new file mode 100644 index 000000000..2bcc39ddd --- /dev/null +++ b/website/content/operations/_index.md @@ -0,0 +1,4 @@ +--- +title: Operations Guide +weight: 60 +--- diff --git a/website/content/pro/_index.md b/website/content/pro/_index.md index eb294cc06..dd39b5ada 100644 --- a/website/content/pro/_index.md +++ b/website/content/pro/_index.md @@ -1,59 +1,6 @@ --- title: Dkron Pro +weight: 40 --- -## Getting started - -Dkron Pro provides a backend store out of the box, based on etcd. - -To configure the storage a sample `etcd.conf.yaml` file is provided in `/etc/dkron` path. Editing the file, allows to configure several options for the embedded store. - -The location of the store configuration can be set in the command line or in the dkron config file `/etc/dkron/dkron.yml` using `etcd-config-file-path` parameter. - -### Starting a single node - -Works out of the box, good for non HA installations. - -- System service: If no changes are done to the default config files, dkron will start as a service in single mode. -- Command line: Running a single node with default config can be done by running: `dkron agent --server` - -Check your server is working: `curl localhost:8080/v1` - -Simple as that, now it is time to add some jobs: - -```bash -curl localhost:8080/v1/jobs -XPOST -d '{ - "name": "job1", - "schedule": "@every 10s", - "timezone": "Europe/Berlin", - "owner": "Platform Team", - "owner_email": "platform@example.com", - "disabled": false, - "tags": { - "dkron_server": "true" - }, - "concurrency": "allow", - "executor": "shell", - "executor_config": { - "command": "date" - } -}` -``` - -## Configuration - -Dkron Pro uses the [same parameters](/basics/configuration) as Dkron OSS and add some extra parameters. - -### Command line options - -* `--etcd-config-file-path` - Etcd node config -* `--username` - Authentication username -* `--password` - Authentication password -* `--cert-file` - Path to the client server TLS cert file -* `--key-file` - Path to the client server TLS key file -* `--client-crl-file` - Path to the client certificate revocation list file -* `--trusted-ca-file` - Path to the client server TLS trusted CA cert file -* `--client-cert-auth` - Enable client cert authentication -* `--auto-tls` - Client TLS using generated certificates - -{{% children style="card" depth="2" description="true" %}} +{{% children style="h3" description="true" %}} diff --git a/website/content/pro/clustering.md b/website/content/pro/clustering.md index 6da9d248e..f97c37571 100644 --- a/website/content/pro/clustering.md +++ b/website/content/pro/clustering.md @@ -4,19 +4,9 @@ title: Clustering ## Configure a cluster -Dkron can run in HA mode, avoiding SPOFs, this mode provides better scalability and better reliability for users that wants a high level of confidence in the cron jobs they need to run. +First follow the Dkron [clustering guide](/usage/clustering) then you can continue with this guide. -To form a cluster, server nodes need to know the address of its peers as in the following example: - -```yaml -# dkron.yml -join: -- 10.19.3.9 -- 10.19.4.64 -- 10.19.7.215 -``` - -On the other side, the embedded store also needs to know its peers, it needs its own configuration as in the following example: +The embedded store also needs to know its peers, it needs its own configuration as in the following example: ```yaml # etcd.conf.yaml @@ -25,3 +15,5 @@ initial-cluster: dkron1=https://10.19.3.9:2380,dkron2=https://10.19.4.64:2380,dk ``` With this configuration Dkron Pro should start in cluster mode with embedded storage. + +For a more in detail guide of clustering with etcd follow this guide: https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/clustering.md diff --git a/website/content/commercial-support.md b/website/content/pro/commercial-support.md similarity index 100% rename from website/content/commercial-support.md rename to website/content/pro/commercial-support.md diff --git a/website/content/pro/configuration.md b/website/content/pro/configuration.md new file mode 100644 index 000000000..3930677d5 --- /dev/null +++ b/website/content/pro/configuration.md @@ -0,0 +1,20 @@ + +--- +title: Configuration +--- + +## Configuration + +Dkron Pro uses the [same parameters](/basics/configuration) as Dkron OSS and add some extra parameters. + +### Command line options + +* `--etcd-config-file-path` - Etcd node config +* `--username` - Authentication username +* `--password` - Authentication password +* `--cert-file` - Path to the client server TLS cert file +* `--key-file` - Path to the client server TLS key file +* `--client-crl-file` - Path to the client certificate revocation list file +* `--trusted-ca-file` - Path to the client server TLS trusted CA cert file +* `--client-cert-auth` - Enable client cert authentication +* `--auto-tls` - Client TLS using generated certificates diff --git a/website/content/pro/ecs.md b/website/content/pro/ecs.md index 286f712c0..38b9b315a 100644 --- a/website/content/pro/ecs.md +++ b/website/content/pro/ecs.md @@ -19,7 +19,9 @@ To allow the ECS Task runner to run tasks, the machine running Dkron needs to ha "ecs:RunTask", "ecs:DescribeTasks", "ecs:DescribeTaskDefinition", - "logs:FilterLogEvents" + "logs:FilterLogEvents", + "logs:DescribeLogStreams", + "logs:PutLogEvents" ], "Resource": [ "*" @@ -34,6 +36,7 @@ To configure a job to be run with the ECS executor: Example using an existing taskdef ```json +{ "executor": "ecs", "executor_config": { "taskdefName": "mytaskdef-family", @@ -43,11 +46,13 @@ Example using an existing taskdef "service": "mycontainer", "overrides": "echo,\"Hello from dkron\"" } +} ``` Example using a provided taskdef ```json +{ "executor": "ecs", "executor_config": { "taskdefBody": "{\"containerDefinitions\": [{\"essential\": true,\"image\": \"hello-world\",\"memory\": 100,\"name\": \"hello-world\"}],\"family\": \"helloworld\"}", @@ -57,6 +62,7 @@ Example using a provided taskdef "env": "ENVIRONMENT=variable", "maxAttempts": 5000 } +} ``` This is the complete list of configuration parameters of the plugin: diff --git a/website/content/pro/quick-start.md b/website/content/pro/quick-start.md new file mode 100644 index 000000000..efc9a2f0a --- /dev/null +++ b/website/content/pro/quick-start.md @@ -0,0 +1,42 @@ +--- +title: Quick start +weight: 10 +--- + +## Getting started + +Dkron Pro provides a clustering backend store out of the box based on etcd. + +To configure the storage a sample `etcd.conf.yaml` file is provided in `/etc/dkron` path. Editing the file, allows to configure several options for the embedded store. + +The location of the store configuration can be set in the command line or in the dkron config file `/etc/dkron/dkron.yml` using `etcd-config-file-path` parameter. + +### Starting a single node + +Works out of the box, good for non HA installations. + +- System service: If no changes are done to the default config files, dkron will start as a service in single mode. +- Command line: Running a single node with default config can be done by running: `dkron agent --server` + +Check your server is working: `curl localhost:8080/v1` + +Simple as that, now it is time to add some jobs: + +```bash +curl localhost:8080/v1/jobs -XPOST -d '{ + "name": "job1", + "schedule": "@every 10s", + "timezone": "Europe/Berlin", + "owner": "Platform Team", + "owner_email": "platform@example.com", + "disabled": false, + "tags": { + "dkron_server": "true" + }, + "concurrency": "allow", + "executor": "shell", + "executor_config": { + "command": "date" + } +}' +``` diff --git a/website/content/products/_index.md b/website/content/products/_index.md index e2f0b33e9..1c886d916 100644 --- a/website/content/products/_index.md +++ b/website/content/products/_index.md @@ -1,5 +1,6 @@ --- title: Products +weight: 70 --- {{% children style="card" depth="2" description="true" %}} diff --git a/website/content/swagger.yaml b/website/content/swagger.yaml index c45d0d3c5..513f54836 100644 --- a/website/content/swagger.yaml +++ b/website/content/swagger.yaml @@ -41,7 +41,9 @@ paths: collectionFormat: multi items: type: string - description: Filter jobs by tags + description: | + Filter jobs by tags. Use this param to return only the desired jobs. Use this format ?tags[foo]=bar&tags[baz]=bar + this will only return the jobs tagged with the specified tags. operationId: getJobs tags: - jobs diff --git a/website/content/usage/_index.md b/website/content/usage/_index.md index 1c872c0c7..8c60cfb34 100644 --- a/website/content/usage/_index.md +++ b/website/content/usage/_index.md @@ -1,6 +1,6 @@ --- title: "Usage" -date: 2017-06-30T14:58:54+02:00 +weight: 30 --- {{% children style="card" depth="2" description="true" %}} diff --git a/website/content/usage/clustering.md b/website/content/usage/clustering.md new file mode 100644 index 000000000..59b5a8a58 --- /dev/null +++ b/website/content/usage/clustering.md @@ -0,0 +1,21 @@ +--- +title: Clustering +--- + +## Configure a cluster + +Dkron can run in HA mode, avoiding SPOFs, this mode provides better scalability and better reliability for users that wants a high level of confidence in the cron jobs they need to run. + +To form a cluster, server nodes need to know the address of its peers as in the following example: + +```yaml +# dkron.yml +join: +- 10.19.3.9 +- 10.19.4.64 +- 10.19.7.215 +``` + +### Etcd + +For a more in detail guide of clustering with etcd follow this guide: https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/clustering.md diff --git a/website/content/usage/cron-spec.md b/website/content/usage/cron-spec.md index b9ad454c5..1bd318cff 100644 --- a/website/content/usage/cron-spec.md +++ b/website/content/usage/cron-spec.md @@ -1,6 +1,6 @@ --- title: Cron spec -toc: true +weight: 20 --- ## CRON Expression Format diff --git a/website/content/internal/_index.md b/website/content/usage/internals.md similarity index 100% rename from website/content/internal/_index.md rename to website/content/usage/internals.md diff --git a/website/content/usage/plugins/develop.md b/website/content/usage/plugins/develop.md index f9de6132b..9ba3d6304 100644 --- a/website/content/usage/plugins/develop.md +++ b/website/content/usage/plugins/develop.md @@ -15,7 +15,7 @@ Create a new Go project somewhere in your `$GOPATH`. If you're a GitHub user, we With the directory made, create a main.go file. This project will be a binary so the package is "main": -``` +```go package main import ( diff --git a/website/content/usage/plugins/executors.md b/website/content/usage/plugins/executors.md index 2e75dcfb6..7c5bdbc63 100644 --- a/website/content/usage/plugins/executors.md +++ b/website/content/usage/plugins/executors.md @@ -16,9 +16,11 @@ New plugins will be added, or you can create new ones, to perform different task Add this to a job definition to use the `shell` executor: ```json -"executor": "shell", -"executor_config": { - "command": "echo \"Hello from dkron\"" +{ + "executor": "shell", + "executor_config": { + "command": "echo \"Hello from dkron\"" + } } ``` diff --git a/website/content/usage/target-nodes-spec.md b/website/content/usage/target-nodes-spec.md index 0de636de7..a2396e761 100644 --- a/website/content/usage/target-nodes-spec.md +++ b/website/content/usage/target-nodes-spec.md @@ -1,6 +1,6 @@ --- title: Target nodes spec -toc: true +weight: 10 --- ## Target nodes spec diff --git a/website/layouts/partials/gheader.html b/website/layouts/partials/gheader.html index 6236ba15f..30888e9e1 100644 --- a/website/layouts/partials/gheader.html +++ b/website/layouts/partials/gheader.html @@ -1,40 +1,44 @@ - - - - - - - - - {{ .Title }} :: {{ .Site.Title }} + + + + - - - + + + - - + - - - - - - - + diff --git a/website/themes/docdock b/website/themes/docdock index b667496cb..c4fad2015 160000 --- a/website/themes/docdock +++ b/website/themes/docdock @@ -1 +1 @@ -Subproject commit b667496cb368fa3336c5b2ef310abb74c2f365da +Subproject commit c4fad20156a2089ef3112665aab57c275978abaf