Skip to content

Commit

Permalink
Docs updates for v0.7 (argoproj#159)
Browse files Browse the repository at this point in the history
* Fixing deactivate route bug in webhook gateway

* Fixing deactivate route bug in storage gateway

* fixing event source deactivation -> activation bug

* Started adding support for NATS streaming

* Started working on supporting nats standard and streaming for communication between gateway and sensor

* Changing nats subscription to queue subscription

* Fire K8s event on nats connection/subscription updates

* Updating gateway and sensor controller validate method

* Fixing nats-streaming issue

* Fix storage grid channel issue

* Removing premature return in transformer

* Renaming dispatchProtocol to eventProtocol

* Moving common structs in apis package

* Adding deepcopy and proto files for common api

* making trigger errors more explicit

* Updating docs for v0.7

* Updating docs for v0.7

* Updating docs for v0.7

* Updating README

* Updating gateway installation docs

* Updating makefile

* Updating makefile
  • Loading branch information
VaibhavPage authored and magaldima committed Jan 29, 2019
1 parent 0eda463 commit 7b7e604
Show file tree
Hide file tree
Showing 110 changed files with 1,908 additions and 2,921 deletions.
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Changelog

## v0.7
+ Refactor gateways #147
+ Fixed sensor error state recovery bug #145
+ Ability to add annotations on sensor and gateway pods #143
+ Github gateway
+ Added support for NATS standard and streaming as communication protocol between gateway
and sensor #99

## v0.6
+ Gitlab Gateway #120
+ If sensor is repeatable then deploy it as deployment else job #109
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ make all
Follow the rest of the [quickstart guide](docs/quickstart.md) to install components.

## Changing Types
If you're making a change to the `pkg/apis/sensor/v1alpha1` package, please ensure you re-run the K8 code-generator scripts found in the `/hack` folder. First, ensure you have the `generate-groups.sh` script at the path: `vendor/k8s.io/code-generator/`. Next run the following commands in order:
If you're making a change to the `pkg/apis` package, please ensure you re-run the K8 code-generator scripts found in the `/hack` folder. First, ensure you have the `generate-groups.sh` script at the path: `vendor/k8s.io/code-generator/`. Next run the following commands in order:
```
$ make codegen
```
12 changes: 6 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -194,33 +194,33 @@ mqtt-image: mqtt-linux

# Custom gateways
storage-grid:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags '${LDFLAGS}' -o ${DIST_DIR}/storagegrid-gateway ./gateways/custom/storagegrid/cmd
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags '${LDFLAGS}' -o ${DIST_DIR}/storagegrid-gateway ./gateways/community/storagegrid/cmd

storage-grid-linux:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 make storage-grid

storage-grid-image: storage-grid-linux
docker build -t $(IMAGE_PREFIX)storage-grid-gateway:$(IMAGE_TAG) -f ./gateways/custom/storagegrid/Dockerfile .
docker build -t $(IMAGE_PREFIX)storage-grid-gateway:$(IMAGE_TAG) -f ./gateways/community/storagegrid/Dockerfile .
@if [ "$(DOCKER_PUSH)" = "true" ] ; then docker push $(IMAGE_PREFIX)storage-grid-gateway:$(IMAGE_TAG) ; fi

gitlab:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags '${LDFLAGS}' -o ${DIST_DIR}/gitlab-gateway ./gateways/custom/gitlab/cmd
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags '${LDFLAGS}' -o ${DIST_DIR}/gitlab-gateway ./gateways/community/gitlab/cmd

gitlab-linux:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 make gitlab

gitlab-image: gitlab-linux
docker build -t $(IMAGE_PREFIX)gitlab-gateway:$(IMAGE_TAG) -f ./gateways/custom/gitlab/Dockerfile .
docker build -t $(IMAGE_PREFIX)gitlab-gateway:$(IMAGE_TAG) -f ./gateways/community/gitlab/Dockerfile .
@if [ "$(DOCKER_PUSH)" = "true" ] ; then docker push $(IMAGE_PREFIX)gitlab-gateway:$(IMAGE_TAG) ; fi

github:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags '${LDFLAGS}' -o ${DIST_DIR}/github-gateway ./gateways/custom/github/cmd
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags '${LDFLAGS}' -o ${DIST_DIR}/github-gateway ./gateways/community/github/cmd

github-linux:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 make github

github-image: github-linux
docker build -t $(IMAGE_PREFIX)github-gateway:$(IMAGE_TAG) -f ./gateways/custom/github/Dockerfile .
docker build -t $(IMAGE_PREFIX)github-gateway:$(IMAGE_TAG) -f ./gateways/community/github/Dockerfile .
@if [ "$(DOCKER_PUSH)" = "true" ] ; then docker push $(IMAGE_PREFIX)github-gateway:$(IMAGE_TAG) ; fi

test:
Expand Down
44 changes: 35 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@
[![Go Report Card](https://goreportcard.com/badge/github.com/argoproj/argo-events)](https://goreportcard.com/report/github.com/argoproj/argo-events)

## What is Argo Events?
Argo Events is an open source event-based dependency manager for Kubernetes. The core concept of the project are
* `gateways` which are implemented as a Kubernetes-native Custom Resource Definition that either produce the events internally or process the events that originate from outside the gateways
Argo Events is an event-based dependency manager for Kubernetes. The core concept of the project are
* `gateways` which are implemented as a Kubernetes-native Custom Resource Definition that process events from event source.

* `sensors` which are implemented as a Kubernetes-native Custom Resource Definition that define a set of dependencies and trigger actions.
* `sensors` which are implemented as a Kubernetes-native Custom Resource Definition that define a set of event dependencies and trigger actions.

- Define multiple dependencies from a variety of gateway sources
- Define multiple dependencies from a variety of gateway event sources
- Build custom gateways to support business-level constraint logic
- Trigger messages and Kubernetes object creation after successful dependency resolution
- Trigger Kubernetes object creation after successful dependency resolution
- Trigger escalation after errors, or dependency constraint failures
- Build and manage a distributed, cross-team, event-driven architecture
- Easily leverage Kubernetes-native APIs to monitor dependencies
Expand All @@ -23,10 +23,36 @@ Argo Events is an open source event-based dependency manager for Kubernetes. The
- Scalable & Resilient.
- Simple or Complex dependencies. Manage everything from simple, linear, real-time dependencies to complex, multi-source, batch job dependencies.

<br/>
<br/>

![](docs/architecture.png)

<br/>
<br/>

## Getting Started
[![asciicast](https://asciinema.org/a/207973.png)](https://asciinema.org/a/207973)

- [Installation](./docs/quickstart.md)
- [Guide](./docs/tutorial.md)
- [Want to contribute or develop/run locally?](./CONTRIBUTING.md)
- See where the project is headed in the [roadmap](./ROADMAP.md)
<br/>
<br/>

1. [Installation](./docs/quickstart.md)
2. [Sensor and gateway controllers](docs/controllers-guide.md)
3. [Learn about gateways](docs/gateway-guide.md)
4. [Learn about sensors](docs/sensor-guide.md)
5. [Learn about triggers](docs/trigger-guide.md)
6. Install Gateways and Sensors
1. [Webhook](gateways/core/webhook/install.md)
2. [Artifact](gateways/core/artifact/install.md)
3. [Calendar](gateways/core/calendar/install.md)
4. [Resource](gateways/core/resource/install.md)
5. [File](gateways/core/file/install.md)
6. Streams
1. [NATS](gateways/core/stream/nats/install.md)
2. [KAFKA](gateways/core/stream/kafka/install.md)
3. [AMQP](gateways/core/stream/amqp/install.md)
4. [MQTT](gateways/core/stream/mqtt/install.md)
7. [Write your own gateway](docs/custom-gateway.md)
8. [Want to contribute or develop/run locally?](./CONTRIBUTING.md)
9. See where the project is headed in the [roadmap](./ROADMAP.md)
3 changes: 1 addition & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
# Roadmap

## Development Phase
- Boolean logic for event dependencies in sensor
- E2E testing
- Add NATS Streaming as internal pub-sub system
- Add SNS & SQS gateway Support

## Idea Phase
- Use [hashicorp/go-getter](https://github.com/hashicorp/go-getter) for the `ArtifactReader` interface.
Expand Down
Binary file modified docs/architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
60 changes: 7 additions & 53 deletions docs/artifact-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,60 +6,14 @@ Inlined artifacts are included directly within the sensor resource and decoded a

## S3
Amazon Simple Storage Service (S3) is a block/file/object store for the internet. The standardized [API](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) allows storage and retrieval of data at any time from anywhere on the web. There are a number of S3 providers that include, but are not limited to:
* [Amazon S3](https://aws.amazon.com/s3/?nc2=h_m1)
* [Minio](https://minio.io/)
* [NetApp](https://www.netapp.com/us/products/data-management-software/object-storage-grid-sds.aspx)
* [CEPH](http://docs.ceph.com/docs/master/radosgw/s3/)
* [Rook](https://rook.io/)

### Minio
Argo Events uses the [minio-go](https://github.com/minio/minio-go) client for access to any Amazon S3 compatible object store. [Minio](https://www.minio.io/) is an distributed object storage server. Follow the Minio [Bucket Notification Guide](https://docs.minio.io/docs/minio-bucket-notification-guide) for help with configuring your minio server to listen and monitor for bucket event notifications. Note that you will need to setup a supported message queue for configuring your notification targets (i.e. NATS, WebHooks, Kafka, etc.).

#### Installation on Kubernetes
The [Minio Deployment Quickstart Guide](https://docs.minio.io/docs/minio-deployment-quickstart-guide.html) is useful for help in getting Minio up & running on your orchestration platform. We've also outlined additional steps below to use Minio for signal notifications and as an object store for trigger resources.

1. Install the Helm chart
```
$ helm init
...
$ helm install stable/minio --name artifacts --set service.type=LoadBalancer
...
$ #Verify that the minio pod, the minio service and minio secret are present
$ kubectl get all -n default -l app=minio
NAME READY STATUS RESTARTS AGE
pod/artifacts-minio-85547b6bd9-bhtt8 1/1 Running 0 21m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/artifacts-minio ClusterIP None <none> 9000/TCP 21m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/artifacts-minio 1 1 1 1 21m
NAME DESIRED CURRENT READY AGE
replicaset.apps/artifacts-minio-85547b6bd9 1 1 1 21m
```

2. Create a bucket in Minio and upload the hello-world.yaml into that bucket.
Download the hello-world.yaml from https://raw.githubusercontent.com/argoproj/argo/master/examples/hello-world.yaml
```
$ kubectl port-forward `kubectl get pod -l app=minio -o name` 9000:9000
```
Open the browser at http://localhost:9000
Create a new bucket called 'workflows'.
Upload the hello-world.yaml into that bucket

Argo Events uses the [minio-go](https://github.com/minio/minio-go) client for access to any Amazon S3 compatible object store. [Minio](https://www.minio.io/) is an distributed object storage server. Follow the Minio [Name Notification Guide](https://docs.minio.io/docs/minio-bucket-notification-guide) for help with configuring your minio server to listen and monitor for bucket event notifications. Note that you will need to setup a supported message queue for configuring your notification targets (i.e. NATS, WebHooks, Kafka, etc.).

#### Enabling bucket notifications
Once the Minio server is configured with a notification target and you have restarted the server to put the changes into effect, you now need to explicitely enable event notifications for a specified bucket. Enabling these notifications are out of scope of Argo Events since bucket notifications are a construct within Minio that exists at the `bucket` level. To avoid multiple sensors on the same S3 bucket conflicting with each other, creating, updating, and deleting Minio bucket notifications should be delegated to a separate process with knowledge of all notification targets including those outside of the Argo Events.
```
$ k edit configmap artifacts-minio
$ k delete pod artifacts-minio
```
## File
Artifacts are defined in a file that is mounted via a [PersistentVolume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) within the `sensor-controller` pod.

## File (future enhancement)
This will enable access to file artifacts via a filesystem mounted as a [PersistentVolume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) within the `sensor-controller` pod.
## URL
Artifacts are accessed from web via RESTful API.

## URL (future enhancement)
This will enable access to web artifacts via RESTful API.
## Configmap
Artifact stored in Kubernetes configmap are accessed using the key.
73 changes: 3 additions & 70 deletions docs/controllers-guide.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## Controllers

* Sensor and Gateway controller are the components which manage Sensor and Gateway resources.
* Sensor and Gateway controllers are the components which manage Sensor and Gateway resources.
* Sensor and Gateway are Kubernetes Custom Resources. For more information on K8 CRDs visit, https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/


Expand All @@ -20,9 +20,9 @@ data:
namespace: my-custom-namespace # optional
```
If you don't provide namespace, controller will watch all namespaces for gateway resource.
<b>`namespace`</b>: If you don't provide namespace, controller will watch all namespaces for gateway resource.

<b>Note on `instance-id`</b>: it is used to map a gateway or sensor to a controller.
<b>`instance-id`</b>: it is used to map a gateway or sensor to a controller.
e.g. when you create a gateway with label `gateways.argoproj.io/gateway-controller-instanceid: argo-events`, a
controller with label `argo-events` will process that gateway. `instance-id` for controller are managed using [controller-configmap](https://raw.githubusercontent.com/argoproj/argo-events/master/hack/k8s/manifests/gateway-controller-configmap.yaml)
Basically `instance-id` is used to horizontally scale controllers, so you won't end up overwhelming a controller with large
Expand All @@ -32,73 +32,6 @@ Basically `instance-id` is used to horizontally scale controllers, so you won't

### Gateway controller
Gateway controller watches gateway resource and manages lifecycle of a gateway.
```yaml
# The gateway-controller listens for changes on the gateway CRD and creates gateway
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: gateway-controller
spec:
replicas: 1
template:
metadata:
labels:
app: gateway-controller
spec:
serviceAccountName: argo-events-sa
containers:
- name: gateway-controller
image: argoproj/gateway-controller:latest
imagePullPolicy: Always
env:
- name: GATEWAY_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: GATEWAY_CONTROLLER_CONFIG_MAP
value: gateway-controller-configmap
```

* <b>Lifecycle of Gateway in gateway controller</b>

![](gateway-controller-fsm.png)

### Sensor controller
Sensor controller watches sensor resource and manages lifecycle of a sensor.
```yaml
# The sensor-controller listens for changes on the sensor CRD and creates sensor executor jobs
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: sensor-controller
spec:
replicas: 1
template:
metadata:
labels:
app: sensor-controller
spec:
serviceAccountName: argo-events-sa
containers:
- name: sensor-controller
image: argoproj/sensor-controller:latest
imagePullPolicy: Always
env:
- name: SENSOR_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: SENSOR_CONFIG_MAP
value: sensor-controller-configmap
```

* <b>Lifecycle of a Sensor managed by sensor controller</b>

![](sensor-fsm.png)

<br/>

* <b>Lifecycle of Sensor within sensor pod</b>

![](sensor-pod-fsm.png)

Binary file removed docs/core-gateway-style.png
Binary file not shown.
Loading

0 comments on commit 7b7e604

Please sign in to comment.